Costco
Data Engineer - Merchandising
Costco, Dallas, Texas, United States, 75215
Data Engineers are responsible for developing and operationalizing data pipelines/integrations to make data available for consumption (i.e. Reporting, Data Science/Machine Learning, Data APIs, etc.). This includes data ingestion, data transformation, data validation/quality, data pipeline optimization, orchestration; and deploying code to production via CI/CD. The Data Engineer role requires knowledge of software development/programming methodologies, various data sources (Relational Databases, flat files (csv, delimited), APIs, XML, JSON, etc.), data access (SQL, Python, etc.), followed by expertise in data modeling, cloud architectures/platforms, data warehousing, and data lakes. This role also will partner closely with Product Owners, Data Architects, Platform/DevOps Engineers, etc. to design, build, test, implement, and maintain data pipelines.
The Data Engineer is responsible for developing and maintaining data pipelines and/or data integrations across the In-Warehouse Manufacturing stack. This includes data ingestion, data transformation, data validation/quality, data pipeline optimization, orchestration; and engaging with DevOps Engineers during CI/CD. This is a new team at Costco and will be fast paced, highly-visible; supporting the business goals by being an industry leader in this space. This role is focused on data engineering to build and deliver automated data pipelines to and from various data sources, primarily iSeries to and from Microsoft Dynamics 365. The Data Engineer will partner with Product Owners, Engineering, and Data Platform teams as needed to design, build, test, and automate data pipelines that are relied upon across the company as the single source of truth.
Job Duties/Essential Functions
• Develops complex SQL & Python against a variety of data sources.
• Implements streaming data pipelines using event/message-based architectures.
• Demonstrates ability to communicate technical concepts to non-technical audiences both in written and verbal form.
• Works in tandem with Data Architects to align on data architecture requirements provided by the requestor.
• Defines and maintains optimal data pipeline architecture.
• Identifies, designs, and implements internal process improvements: automating manual processes, optimizing data. delivery/orchestration.
• Demonstrates strong understanding with coding and programming concepts to build data pipelines (e.g. data transformation, data quality, data integration, etc.).
• Analyzes data to spot anomalies, trends, and correlate data to ensure Data Quality.
• Develops data pipelines to store data in defined data models/structures.
• Demonstrates strong understanding of data integration techniques and tools (e.g. Extract, Transform, Load (ETL)/Extract, Load, Transform (ELT) tools.
• Demonstrates strong understanding of database storage concepts (data lake, relational databases, NoSQL, Graph, data warehousing).
• Identifies ways to improve data reliability, efficiency, and quality of data management.
• Performs peer review for another Data Engineer's work.
• Builds or extends data models in Dynamics365.
• Integrates architecture and performance optimization.
• Integrates Platform Engineering, Admin., and support.
• Uses Azure DevOps component to build VMs for Secure Agents (i.e. Gateways).
• Develops data mapping (Source to D365 & D365 to source).
• Upstreams/downstreams data flows architecture and design (mainly iSeries, AS400 IBMi as source and destination for data).
• Develops data pipelines to store data in defined data models and structures to be usable within applications or by other business reporting applications or dashboards.
• Identifies ways to improve data reliability, efficiency, and quality of data management.
• Assesses the integrity of data from multiple sources.
• Manages database configuration including installing and upgrading software and maintaining relevant documentation.
• Develops and operationalizes data pipelines to create enterprise certified data sets that are made available for consumption.
• Designs, develops, and implements ETL/ELT processes.
• Uses Azure services such as Azure SQL DW (Synapse), ADLS, Azure Event Hub, Cosmos, Databricks, Delta-Lake to improve and speed up delivery of our data products and services.
• Implements big data and NoSQL solutions by developing scalable data processing platforms to drive high-value insights to the organization.
• Identifies, designs, and implements internal process improvements: automating manual processes, optimizing data delivery.
• Identifies ways to improve data reliability, efficiency and quality of data management.
• Communicates technical concepts to non-technical audiences both in written and verbal form.
• Regular and reliable workplace attendance at your assigned location.
Ability to operate vehicles, equipment or machinery
• Computer, phone, printer, copier, fax
Non-Essential Functions
• Assists in other areas of the department as necessary.
• Assists in other areas of the company as necessary.
Ability to operate vehicles, equipment or machinery
• Same as essential functions
Experience, Skills, Education & Licenses/Certifications Required:
• 4 years' experience engineering and operationalizing data pipelines with large and complex datasets.
• 2 years' hands-on experience with iSeries / IBMi integration.
• 3 years' experience working with Cloud technologies such as ADLS, Azure Databricks, Spark, Azure Synapse, Cosmos DB and other big data technologies.
• Extensive experience working with various data sources (DB2, SQL,Oracle, flat files (csv, delimited), APIs, XML, JSON.
• Middleware solutions, such as IBM ACE.
• Advanced SQL skills required. Solid understanding of relational databases and business data; ability to write complex SQL queries against a variety of data sources.
• 3 years' experience with Data Modeling and ETL.
• Strong understanding of database storage concepts (data lake, relational databases, NoSQL, or Realm). Recommended:
• Azure Certifications.
• Graph QL.
• D365 data entity models.
• Experience implementing data integration techniques such as event/message-based integration (Azure Event Hub) ETL.
• Experience with Git / Azure DevOps.
• Experience delivering data solutions through agile software development methodologies.
• Exposure to the retail industry.
• Excellent verbal and written communication skills.
• Proficient in Google Workspace applications, including Sheets, Docs, Slides, and Gmail.
• Successful internal candidates will have spent one year or more on their current team.
Other Conditions
• Management will review the Job Analysis for this position prior to a job offer.
Required Documents
• Cover Letter
• Resume
• Last two performance reviews
• Attendance records for current year (Do not include absences covered by paid sick/personal time,
FMLA or other protected absences.)
California applicants, please click here to review the Costco Applicant Privacy Notice.
The Data Engineer is responsible for developing and maintaining data pipelines and/or data integrations across the In-Warehouse Manufacturing stack. This includes data ingestion, data transformation, data validation/quality, data pipeline optimization, orchestration; and engaging with DevOps Engineers during CI/CD. This is a new team at Costco and will be fast paced, highly-visible; supporting the business goals by being an industry leader in this space. This role is focused on data engineering to build and deliver automated data pipelines to and from various data sources, primarily iSeries to and from Microsoft Dynamics 365. The Data Engineer will partner with Product Owners, Engineering, and Data Platform teams as needed to design, build, test, and automate data pipelines that are relied upon across the company as the single source of truth.
Job Duties/Essential Functions
• Develops complex SQL & Python against a variety of data sources.
• Implements streaming data pipelines using event/message-based architectures.
• Demonstrates ability to communicate technical concepts to non-technical audiences both in written and verbal form.
• Works in tandem with Data Architects to align on data architecture requirements provided by the requestor.
• Defines and maintains optimal data pipeline architecture.
• Identifies, designs, and implements internal process improvements: automating manual processes, optimizing data. delivery/orchestration.
• Demonstrates strong understanding with coding and programming concepts to build data pipelines (e.g. data transformation, data quality, data integration, etc.).
• Analyzes data to spot anomalies, trends, and correlate data to ensure Data Quality.
• Develops data pipelines to store data in defined data models/structures.
• Demonstrates strong understanding of data integration techniques and tools (e.g. Extract, Transform, Load (ETL)/Extract, Load, Transform (ELT) tools.
• Demonstrates strong understanding of database storage concepts (data lake, relational databases, NoSQL, Graph, data warehousing).
• Identifies ways to improve data reliability, efficiency, and quality of data management.
• Performs peer review for another Data Engineer's work.
• Builds or extends data models in Dynamics365.
• Integrates architecture and performance optimization.
• Integrates Platform Engineering, Admin., and support.
• Uses Azure DevOps component to build VMs for Secure Agents (i.e. Gateways).
• Develops data mapping (Source to D365 & D365 to source).
• Upstreams/downstreams data flows architecture and design (mainly iSeries, AS400 IBMi as source and destination for data).
• Develops data pipelines to store data in defined data models and structures to be usable within applications or by other business reporting applications or dashboards.
• Identifies ways to improve data reliability, efficiency, and quality of data management.
• Assesses the integrity of data from multiple sources.
• Manages database configuration including installing and upgrading software and maintaining relevant documentation.
• Develops and operationalizes data pipelines to create enterprise certified data sets that are made available for consumption.
• Designs, develops, and implements ETL/ELT processes.
• Uses Azure services such as Azure SQL DW (Synapse), ADLS, Azure Event Hub, Cosmos, Databricks, Delta-Lake to improve and speed up delivery of our data products and services.
• Implements big data and NoSQL solutions by developing scalable data processing platforms to drive high-value insights to the organization.
• Identifies, designs, and implements internal process improvements: automating manual processes, optimizing data delivery.
• Identifies ways to improve data reliability, efficiency and quality of data management.
• Communicates technical concepts to non-technical audiences both in written and verbal form.
• Regular and reliable workplace attendance at your assigned location.
Ability to operate vehicles, equipment or machinery
• Computer, phone, printer, copier, fax
Non-Essential Functions
• Assists in other areas of the department as necessary.
• Assists in other areas of the company as necessary.
Ability to operate vehicles, equipment or machinery
• Same as essential functions
Experience, Skills, Education & Licenses/Certifications Required:
• 4 years' experience engineering and operationalizing data pipelines with large and complex datasets.
• 2 years' hands-on experience with iSeries / IBMi integration.
• 3 years' experience working with Cloud technologies such as ADLS, Azure Databricks, Spark, Azure Synapse, Cosmos DB and other big data technologies.
• Extensive experience working with various data sources (DB2, SQL,Oracle, flat files (csv, delimited), APIs, XML, JSON.
• Middleware solutions, such as IBM ACE.
• Advanced SQL skills required. Solid understanding of relational databases and business data; ability to write complex SQL queries against a variety of data sources.
• 3 years' experience with Data Modeling and ETL.
• Strong understanding of database storage concepts (data lake, relational databases, NoSQL, or Realm). Recommended:
• Azure Certifications.
• Graph QL.
• D365 data entity models.
• Experience implementing data integration techniques such as event/message-based integration (Azure Event Hub) ETL.
• Experience with Git / Azure DevOps.
• Experience delivering data solutions through agile software development methodologies.
• Exposure to the retail industry.
• Excellent verbal and written communication skills.
• Proficient in Google Workspace applications, including Sheets, Docs, Slides, and Gmail.
• Successful internal candidates will have spent one year or more on their current team.
Other Conditions
• Management will review the Job Analysis for this position prior to a job offer.
Required Documents
• Cover Letter
• Resume
• Last two performance reviews
• Attendance records for current year (Do not include absences covered by paid sick/personal time,
FMLA or other protected absences.)
California applicants, please click here to review the Costco Applicant Privacy Notice.