Hire Talent
Data Engineer 3
Hire Talent, Beaverton, Oregon, us, 97078
Job Title: Data Engineer 3Job Location: Beaverton OR (Remote)Job Duration: 12 Months on W2
About Us:We are the Consumer Data Engineering team (CoDE) at ***, seeking an experienced Senior Data Engineer to join our team. As a Senior Data Engineer, you will play a critical role in designing, building, and maintaining our big data infrastructure, ensuring the scalability, reliability, and performance of our data systems.
Job Summary:We are looking for a highly skilled Senior Data Engineer with a strong background in big data engineering, cloud computing, and software development. The ideal candidate will have a proven track record of designing and implementing scalable data solutions using AWS, Spark, and Python. The candidate should have hands-on experience with Databricks, optimizing Spark applications, and building ETL pipelines. Experience with CI/CD, unit testing, and big data problem-solving is a plus.
Key Responsibilities:• Design, build, and maintain large-scale data pipelines using AWS EMR, Spark, and Python• Develop and optimize Spark applications and ETL pipelines for performance and scalability• Collaborate with product managers and analysts to design and implement data models and data warehousing solutions• Work with cross-functional teams to integrate data systems with other applications and services• Ensure data quality, integrity, and security across all data systems• Develop and maintain unit test cases for data pipelines and applications• Implement CI/CD pipelines for automated testing and deployment• Collaborate with the DevOps team to ensure seamless deployment of data applications• Stay up to date with industry trends and emerging technologies in big data and cloud computing
Requirements:• At least 5 years of experience in data engineering, big data, or a related field• Proficiency in Spark, including Spark Core, Spark SQL, and Spark Streaming• Experience with AWS EMR, including cluster management and job optimization• Strong skills in Python, including data structures, algorithms, and software design patterns• Hands-on experience with Databricks, including Databricks Lakehouse (advantageous)• Experience with optimizing Spark applications and ETL pipelines for performance and scalability• Good understanding of data modeling, data warehousing, and data governance• Experience with CI/CD tools such as Jenkins, GitLab, or CircleCI (advantageous)• Strong understanding of software development principles, including unit testing and test-driven development• Ability to design and implement scalable data solutions that meet business requirements• Strong problem-solving skills, with the ability to debug complex data issues• Excellent communication and collaboration skills, with the ability to work with cross-functional teams
Nice to Have:• Experience with Databricks Lakehouse• Knowledge of data engineering best practices and design patterns• Experience with agile development methodologies, such as Scrum or Kanban
REQUIRED:5+ years in data engineeringExperience with AWS EMR, including cluster management and job optimizationProficiency in Spark, including Spark Core, Spark SQL, and Spark Streaming
IDEAL CANDIDATE:Hands on experience with Databricks and Lakehouse dataMigrating from AWS EMR to Databricks Lakehouse
Team:Supporting the Commercial Analytics Engineering department
About Us:We are the Consumer Data Engineering team (CoDE) at ***, seeking an experienced Senior Data Engineer to join our team. As a Senior Data Engineer, you will play a critical role in designing, building, and maintaining our big data infrastructure, ensuring the scalability, reliability, and performance of our data systems.
Job Summary:We are looking for a highly skilled Senior Data Engineer with a strong background in big data engineering, cloud computing, and software development. The ideal candidate will have a proven track record of designing and implementing scalable data solutions using AWS, Spark, and Python. The candidate should have hands-on experience with Databricks, optimizing Spark applications, and building ETL pipelines. Experience with CI/CD, unit testing, and big data problem-solving is a plus.
Key Responsibilities:• Design, build, and maintain large-scale data pipelines using AWS EMR, Spark, and Python• Develop and optimize Spark applications and ETL pipelines for performance and scalability• Collaborate with product managers and analysts to design and implement data models and data warehousing solutions• Work with cross-functional teams to integrate data systems with other applications and services• Ensure data quality, integrity, and security across all data systems• Develop and maintain unit test cases for data pipelines and applications• Implement CI/CD pipelines for automated testing and deployment• Collaborate with the DevOps team to ensure seamless deployment of data applications• Stay up to date with industry trends and emerging technologies in big data and cloud computing
Requirements:• At least 5 years of experience in data engineering, big data, or a related field• Proficiency in Spark, including Spark Core, Spark SQL, and Spark Streaming• Experience with AWS EMR, including cluster management and job optimization• Strong skills in Python, including data structures, algorithms, and software design patterns• Hands-on experience with Databricks, including Databricks Lakehouse (advantageous)• Experience with optimizing Spark applications and ETL pipelines for performance and scalability• Good understanding of data modeling, data warehousing, and data governance• Experience with CI/CD tools such as Jenkins, GitLab, or CircleCI (advantageous)• Strong understanding of software development principles, including unit testing and test-driven development• Ability to design and implement scalable data solutions that meet business requirements• Strong problem-solving skills, with the ability to debug complex data issues• Excellent communication and collaboration skills, with the ability to work with cross-functional teams
Nice to Have:• Experience with Databricks Lakehouse• Knowledge of data engineering best practices and design patterns• Experience with agile development methodologies, such as Scrum or Kanban
REQUIRED:5+ years in data engineeringExperience with AWS EMR, including cluster management and job optimizationProficiency in Spark, including Spark Core, Spark SQL, and Spark Streaming
IDEAL CANDIDATE:Hands on experience with Databricks and Lakehouse dataMigrating from AWS EMR to Databricks Lakehouse
Team:Supporting the Commercial Analytics Engineering department