Lead Data Pipeline Engineer - Machine Learning

Lead Data Pipeline Engineer - Machine LearningTechnologyRemote - United States$170,000 - $200,000 + EquityAbout Us:We are working with a fast-growing startup focused on automating the construction industry through cutting-edge AI and workflow tools. The company's leadership includes seasoned entrepreneurs and industry experts, and their software is used by thousands in the field. With strong partnerships among Fortune 500 companies, they are driving real change in a sector full of inefficiencies.They are looking for a hands-on Lead Data Engineer who thrives in a dynamic environment, has a passion for automation, and is eager to work at the intersection of data engineering and machine learning to build innovative solutions.Key Responsibilities:As the Lead Data Engineer - Machine Learning, you will lead the design and development of scalable data pipelines that integrate with machine learning models to enhance automation tools for construction workflows. You'll collaborate closely with the machine learning team, driving innovation and efficiency in how data is processed and utilized. Other responsibilities include:Lead the design, development, and maintenance of a robust file processing pipeline infrastructureOrchestrate the flow of data through various stages of processing and machine learning integrationEnsure observability and monitoring of pipeline health and performanceIntegrate data from multiple sources, including storage platforms, project management tools, and external APIsImplement data quality checks and error-handling mechanisms to ensure data integrityWork closely with the machine learning team to optimize data pipelines and model deploymentRequirements:Proficiency in Python and database systems (SQL, NoSQL)Experience with pipeline orchestration tools (e.g., Prefect), infrastructure-as-code (e.g., Terraform), and observability/monitoring toolsFamiliarity with serverless architectures (e.g., AWS Lambda)Understanding of machine learning workflows and requirementsKnowledge of data modeling and data warehouse conceptsInterest in transforming the construction industry through technologyBenefits:As the Lead Data Engineer - Machine Learning, you can expect $170,000 to $200,000 in compensation, along with Equity, health, dental, and vision benefits.Key Words:Python, SQL, NoSQL, AWS Lambda, Prefect, Terraform, Data Pipelines, ETL, Data Engineering, Machine Learning, Infrastructure, Construction Industry, Startup