Hirewell
Senior Data Engineer
Hirewell, Chicago, Illinois, United States, 60290
Chicago, IL
Direct Hire
Salary Range : $150,000 - $210,000
Position Overview
We are looking for an experienced Senior Data Engineer to join our data platform team. This role involves developing efficient data systems to support our analytics engine. The ideal candidate will be proficient in Python and SQL, with a strong background in data analytics. You will collaborate closely with implementation teams, data scientists, and product stakeholders to advance this innovative product.
Key Responsibilities
Develop scalable batch data processing frameworks for analytical transformationsGenerate foundational data for machine learning models and deploy them into productionEnhance the performance of data pipelines and database queriesMonitor and resolve data quality issuesQualifications
5-10 years of experience with:
PythonSQLApache AirflowSnowflakeAWS services (RDS, EMR, S3)Git / GitHubSpark (PySpark)
Extensive background in building data processing systems and analytical data productsKnowledge of machine learning algorithms and data analysis techniquesExperience and Skills
Python ExpertiseAdvanced proficiency in Python for data engineering, with over 5 years of experience developing optimized ETL pipelines for data transformation and analysis. Skilled in using Pandas, NumPy, and SQLAlchemy for data manipulation and database interaction.Proven ability to design and execute complex data models and ETL processes in Snowflake, ensuring high performance and scalability. Experience with Python-based frameworks for analytics and machine learning is a plus.Experience integrating Python applications with cloud services and data orchestration tools, such as AWS and Apache Airflow.
Batch Data ProcessingExperience in designing and implementing batch data processing pipelines (ETL/ELT) to extract, transform, and load data.
Snowflake ExperienceProficient in designing and implementing data models in Snowflake, optimizing for performance and scalability. Familiar with Snowflake's architecture, including data sharing and warehousing features.Experience with Snowpark and Snowpipe is advantageous.
AWS ProficiencyExtensive experience with AWS services (RDS, EMR, S3), including provisioning and management through Boto3, CDK, CLI, or Apache Airflow operators.
Spark (PySpark)Experience with Spark (PySpark) is a plus.
Version ControlProficient in using Git / GitHub, with experience in PyCharm or similar IDEs.
Apache AirflowExperience building DAGs for ETL/ELT, including single-use case DAGs and reusable frameworks.
SQL ExpertiseAdvanced SQL skills, including optimization techniques such as explain plans, join optimization, indexing, and advanced functions.
Direct Hire
Salary Range : $150,000 - $210,000
Position Overview
We are looking for an experienced Senior Data Engineer to join our data platform team. This role involves developing efficient data systems to support our analytics engine. The ideal candidate will be proficient in Python and SQL, with a strong background in data analytics. You will collaborate closely with implementation teams, data scientists, and product stakeholders to advance this innovative product.
Key Responsibilities
Develop scalable batch data processing frameworks for analytical transformationsGenerate foundational data for machine learning models and deploy them into productionEnhance the performance of data pipelines and database queriesMonitor and resolve data quality issuesQualifications
5-10 years of experience with:
PythonSQLApache AirflowSnowflakeAWS services (RDS, EMR, S3)Git / GitHubSpark (PySpark)
Extensive background in building data processing systems and analytical data productsKnowledge of machine learning algorithms and data analysis techniquesExperience and Skills
Python ExpertiseAdvanced proficiency in Python for data engineering, with over 5 years of experience developing optimized ETL pipelines for data transformation and analysis. Skilled in using Pandas, NumPy, and SQLAlchemy for data manipulation and database interaction.Proven ability to design and execute complex data models and ETL processes in Snowflake, ensuring high performance and scalability. Experience with Python-based frameworks for analytics and machine learning is a plus.Experience integrating Python applications with cloud services and data orchestration tools, such as AWS and Apache Airflow.
Batch Data ProcessingExperience in designing and implementing batch data processing pipelines (ETL/ELT) to extract, transform, and load data.
Snowflake ExperienceProficient in designing and implementing data models in Snowflake, optimizing for performance and scalability. Familiar with Snowflake's architecture, including data sharing and warehousing features.Experience with Snowpark and Snowpipe is advantageous.
AWS ProficiencyExtensive experience with AWS services (RDS, EMR, S3), including provisioning and management through Boto3, CDK, CLI, or Apache Airflow operators.
Spark (PySpark)Experience with Spark (PySpark) is a plus.
Version ControlProficient in using Git / GitHub, with experience in PyCharm or similar IDEs.
Apache AirflowExperience building DAGs for ETL/ELT, including single-use case DAGs and reusable frameworks.
SQL ExpertiseAdvanced SQL skills, including optimization techniques such as explain plans, join optimization, indexing, and advanced functions.