Data Engineer Jobs
Walmart Senior Data Engineer - Sunnyvale - sql scala apache kafka Agriculture
Data Engineer Jobs, Sunnyvale, California, United States, 94087
What you'll do...Position:
Senior Data EngineerJob Location:
680 W. California Avenue, Sunnyvale, CA 94086Duties:Build scalable, high performance and robust Java, Scala, and Python applications.Build complex large scale data pipelines using cutting-edge big data technologies such as Spark, Kafka, Apache Airflow, Apache NiFi, Hadoop, and Hive which are used for customer interactions and customer transactions.Responsible for designing, developing, and maintaining ETL data processing pipelines on the Google Cloud Platform for customer profile and engagement data sets.Discover batch data from external sources to ingest it into the team's data pipeline and store it in the organization's HDFS space.Work on user engagement data such as clicks, email interactions and engagements, and push notification engagements.Collect, clean, organize and analyze big data of user interactions with platforms such as mobile app and website and store the data in Hive tables to assist in marketing.Work on multiple projects to ensure that the collected customer interaction data meets data quality standards and rules.Collect and visualize data metrics using Grafana, Scala, and MySQL.Configure, maintain, and optimize GCP cloud compute infrastructure including migration of workflows to reduce cloud costs.Implement CICD (Continuous Integration and Continuous Development) for the platform using Jenkins, GitHub, and the GCP console.Actively work with product teams to plan out development cycle for new features via JIRA tickets.Help analyze A/B test results and plan out a test-driven development approach as per AGILE methodology.Participate in the discovery phase and build high level designs.Evaluate and fine-tune systems for speed, robustness and cost efficiency.Troubleshoot production issues as necessary by responding to pager duty alerts triggered by Xmatters.Participate in high-impact tech initiatives from end-to-end, covering all stages of the software development lifecycle and all the components of the solution.Apply security patches, upgrading versions, renewing certificates, and managing the change management process.Provide new developers with training, a system overview, and deep dive presentations for knowledge transfer.Minimum education and experience required:Master's degree or equivalent in Computer Science, Engineering (Any), or related field and 1 year of experience in large scale enterprise software development environment; OR Bachelor's degree or equivalent in Computer Science, Engineering (Any), or related field and 4 years of experience in large scale enterprise software development environment.Skills required:Experience architecting and developing complex and highly available services over Cloud Infrastructure using Java, Python and Cloud Computing technologies.Experience storing and processing large-scale customer data using SQL, NoSQL databases and Data Warehouse services.Experience utilizing Agile processes including scrum, backlog grooming, sprint planning, demo, and retro for software development.Experience implementing end-to-end CI/CD pipelines using Jenkins pipeline and Github for version control.Experience designing and implementing complex workflow management for data pipelines.Experience developing and emitting metrics to build analysis dashboards to monitor data pipeline performance using Cloud Metrics technologies including AWS CloudWatch.Experience creating comprehensive test strategies with JUnit, Mockito for unit tests, integration tests and code coverage.Experience creating high level design documents and implementation plans for end to end project delivery.Experience generating alerts on various metrics and data counts to trigger notifications to help maintain the availability and health of the system.Experience staying updated on emerging data engineering technologies including emerging Cloud Data Storage technologies and Java libraries.Experience implementing Identity and Access Management to manage groups and permissions to access resources.Salary Range:
$174,518/year to $234,000/year. Additional compensation includes annual or quarterly performance incentives. Additional compensation for certain positions may also include: Regional Pay Zone (RPZ) (based on location) and Stock equity incentives.Benefits:At Walmart, we offer competitive pay as well as performance-based incentive awards and other great benefits for a happier mind, body, and wallet.Health benefits include medical, vision and dental coverage.Financial benefits include 401(k), stock purchase and company-paid life insurance.Paid time off benefits include PTO (including sick leave), parental leave, family care leave, bereavement, jury duty and voting.Other benefits include short-term and long-term disability, education assistance with 100% company paid college degrees, company discounts, military service pay, adoption expense reimbursement, and more.Eligibility requirements apply to some benefits and may depend on your job classification and length of employment. Benefits are subject to change and may be subject to a specific plan or program terms.
#J-18808-Ljbffr
Senior Data EngineerJob Location:
680 W. California Avenue, Sunnyvale, CA 94086Duties:Build scalable, high performance and robust Java, Scala, and Python applications.Build complex large scale data pipelines using cutting-edge big data technologies such as Spark, Kafka, Apache Airflow, Apache NiFi, Hadoop, and Hive which are used for customer interactions and customer transactions.Responsible for designing, developing, and maintaining ETL data processing pipelines on the Google Cloud Platform for customer profile and engagement data sets.Discover batch data from external sources to ingest it into the team's data pipeline and store it in the organization's HDFS space.Work on user engagement data such as clicks, email interactions and engagements, and push notification engagements.Collect, clean, organize and analyze big data of user interactions with platforms such as mobile app and website and store the data in Hive tables to assist in marketing.Work on multiple projects to ensure that the collected customer interaction data meets data quality standards and rules.Collect and visualize data metrics using Grafana, Scala, and MySQL.Configure, maintain, and optimize GCP cloud compute infrastructure including migration of workflows to reduce cloud costs.Implement CICD (Continuous Integration and Continuous Development) for the platform using Jenkins, GitHub, and the GCP console.Actively work with product teams to plan out development cycle for new features via JIRA tickets.Help analyze A/B test results and plan out a test-driven development approach as per AGILE methodology.Participate in the discovery phase and build high level designs.Evaluate and fine-tune systems for speed, robustness and cost efficiency.Troubleshoot production issues as necessary by responding to pager duty alerts triggered by Xmatters.Participate in high-impact tech initiatives from end-to-end, covering all stages of the software development lifecycle and all the components of the solution.Apply security patches, upgrading versions, renewing certificates, and managing the change management process.Provide new developers with training, a system overview, and deep dive presentations for knowledge transfer.Minimum education and experience required:Master's degree or equivalent in Computer Science, Engineering (Any), or related field and 1 year of experience in large scale enterprise software development environment; OR Bachelor's degree or equivalent in Computer Science, Engineering (Any), or related field and 4 years of experience in large scale enterprise software development environment.Skills required:Experience architecting and developing complex and highly available services over Cloud Infrastructure using Java, Python and Cloud Computing technologies.Experience storing and processing large-scale customer data using SQL, NoSQL databases and Data Warehouse services.Experience utilizing Agile processes including scrum, backlog grooming, sprint planning, demo, and retro for software development.Experience implementing end-to-end CI/CD pipelines using Jenkins pipeline and Github for version control.Experience designing and implementing complex workflow management for data pipelines.Experience developing and emitting metrics to build analysis dashboards to monitor data pipeline performance using Cloud Metrics technologies including AWS CloudWatch.Experience creating comprehensive test strategies with JUnit, Mockito for unit tests, integration tests and code coverage.Experience creating high level design documents and implementation plans for end to end project delivery.Experience generating alerts on various metrics and data counts to trigger notifications to help maintain the availability and health of the system.Experience staying updated on emerging data engineering technologies including emerging Cloud Data Storage technologies and Java libraries.Experience implementing Identity and Access Management to manage groups and permissions to access resources.Salary Range:
$174,518/year to $234,000/year. Additional compensation includes annual or quarterly performance incentives. Additional compensation for certain positions may also include: Regional Pay Zone (RPZ) (based on location) and Stock equity incentives.Benefits:At Walmart, we offer competitive pay as well as performance-based incentive awards and other great benefits for a happier mind, body, and wallet.Health benefits include medical, vision and dental coverage.Financial benefits include 401(k), stock purchase and company-paid life insurance.Paid time off benefits include PTO (including sick leave), parental leave, family care leave, bereavement, jury duty and voting.Other benefits include short-term and long-term disability, education assistance with 100% company paid college degrees, company discounts, military service pay, adoption expense reimbursement, and more.Eligibility requirements apply to some benefits and may depend on your job classification and length of employment. Benefits are subject to change and may be subject to a specific plan or program terms.
#J-18808-Ljbffr