Compunnel
Senior Data Engineer
Compunnel, Bellevue, Washington, us, 98009
Job Summary
Client is looking for a Sr. Data Engineer with extensive experience in Azure, Databricks, PowerShell, and SQL to contribute to building scalable data engineering solutions. This role involves working within an Azure + Databricks Lakehouse environment, utilizing a variety of tools and technologies to optimize data pipelines, implement DevOps practices, and improve the performance and efficiency of data applications.
Key Responsibilities
• Develop, maintain, and optimize data pipelines leveraging Azure Data Factory (ADF), Databricks, and PySpark
• Design and implement CI/CD pipelines using Azure DevOps
• Utilize SQL (TSQL, PostgreSQL) and PowerShell for scripting and automation
• Optimize code for performance, including reading DAGs and resolving CBO issues
• Ensure consistency and repeatability in code deployment using custom SQL frameworks
• Develop and maintain data lakes and data warehouses using ADLS, Delta Lake, and Parquet formats
• Participate in peer code reviews and maintain adherence to SDLC, including DevOps practices
• Advise and support the team on data modeling, performance tuning, and system scalability
• Collaborate with cross-functional teams and clarify technical requirements with customers
• Manage defect tracking and resolution, providing proactive measures to improve code quality
• Estimate time and resources required for developing/debugging data features/components
Required Qualifications
• 8+ years of experience in Azure, Databricks, and related data engineering technologies
• Strong proficiency with Azure Data Factory (ADF), ADLS, and Azure DevOps
• Experience with Databricks, PySpark, and SparkSQL
• Proficiency with SQL (TSQL, PostgreSQL) and PowerShell scripting
• Familiarity with data formats like Parquet and Delta Lake
• Deep understanding of SDLC and CI/CD practices
• Expertise in optimizing code for performance and scalability
• Strong problem-solving skills and ability to break down complex challenges
• Excellent communication and collaboration skills in an Agile environment
Preferred Qualifications
• Experience with big data tools and technologies (e.g., Hadoop, Kafka, etc.)
• Experience with cloud-based data solutions and architecture design
• Familiarity with Agile methodologies (Scrum, Kanban)
• Strong experience with performance tuning, indexing, and partitioning
• Certification in Azure Data Engineering or related domain
Certifications
• Azure Data Engineer Associate (preferred)
• Databricks Certified Associate Developer for Apache Spark (preferred)
Location:
Education:
Bachelors Degree
Client is looking for a Sr. Data Engineer with extensive experience in Azure, Databricks, PowerShell, and SQL to contribute to building scalable data engineering solutions. This role involves working within an Azure + Databricks Lakehouse environment, utilizing a variety of tools and technologies to optimize data pipelines, implement DevOps practices, and improve the performance and efficiency of data applications.
Key Responsibilities
• Develop, maintain, and optimize data pipelines leveraging Azure Data Factory (ADF), Databricks, and PySpark
• Design and implement CI/CD pipelines using Azure DevOps
• Utilize SQL (TSQL, PostgreSQL) and PowerShell for scripting and automation
• Optimize code for performance, including reading DAGs and resolving CBO issues
• Ensure consistency and repeatability in code deployment using custom SQL frameworks
• Develop and maintain data lakes and data warehouses using ADLS, Delta Lake, and Parquet formats
• Participate in peer code reviews and maintain adherence to SDLC, including DevOps practices
• Advise and support the team on data modeling, performance tuning, and system scalability
• Collaborate with cross-functional teams and clarify technical requirements with customers
• Manage defect tracking and resolution, providing proactive measures to improve code quality
• Estimate time and resources required for developing/debugging data features/components
Required Qualifications
• 8+ years of experience in Azure, Databricks, and related data engineering technologies
• Strong proficiency with Azure Data Factory (ADF), ADLS, and Azure DevOps
• Experience with Databricks, PySpark, and SparkSQL
• Proficiency with SQL (TSQL, PostgreSQL) and PowerShell scripting
• Familiarity with data formats like Parquet and Delta Lake
• Deep understanding of SDLC and CI/CD practices
• Expertise in optimizing code for performance and scalability
• Strong problem-solving skills and ability to break down complex challenges
• Excellent communication and collaboration skills in an Agile environment
Preferred Qualifications
• Experience with big data tools and technologies (e.g., Hadoop, Kafka, etc.)
• Experience with cloud-based data solutions and architecture design
• Familiarity with Agile methodologies (Scrum, Kanban)
• Strong experience with performance tuning, indexing, and partitioning
• Certification in Azure Data Engineering or related domain
Certifications
• Azure Data Engineer Associate (preferred)
• Databricks Certified Associate Developer for Apache Spark (preferred)
Location:
Education:
Bachelors Degree