Data Pipeline Engineer

DATA PIPELINE ENGINEER GEN AI STARTUP NEW YORK, NY (HYBRID) $170,000 - $200,000 THE COMPANY We are working with a generative AI startup that has recently raised its next round of funding and is scaling like crazy. This innovative startup within the construction industry is looking to hire a Data Pipeline Engineer to design, develop, and maintain pipeline infrastructure and workflows while working closely with the ML team. THE ROLE - DATA PIPELINE ENGINEER Design, develop and maintain robust file processing pipeline infrastructure Orchestrate the flow of data through various stages of processing Ensure observability and monitoring of the pipeline's health Integrate data from various sources including industry storage platforms, project management tools, and external APIs Implement data quality checks and error handling mechanisms to ensure data integrity Collaborate with the machine learning team to enhance pipeline functionality and efficiency SKILLS AND REQUIREMENTS Experience in ETL development and data engineering Expert coding proficiency in Python and database systems (SQL, noSQL) Strong experience with pipeline orchestration tools (eg. Prefect), infrastructure-as-code (eg. Terraform), and observability and monitoring tools Understanding of serverless architectures (eg. AWS Lambda) Familiarity with ML workflows and requirements (to effectively collaborate with the ML team) Knowledge of data modeling and data warehouse concepts Startup experience THE BENEFITS Data Pipeline Engineer role offers a competitive salary range of $170,000- $200,000, depending on experience, along with equity in the company. Comprehensive health, dental, and vision benefits are also included. HOW TO APPLY Please register your interest by sending your resume to Malia Jalbert via the Apply link on this page. KEYWORDS Data Pipelines, Engineer, ETL, Data Engineering, Artificial Intelligence, Machine Learning, Python, Amazon Web Services, AWS, Data Warehouse, Terraform, Prefect, SQL, NoSQL, Data Warehouse