Inizio Partners
Data Engineer, Gen AI
Inizio Partners, New York, New York, us, 10261
About the job Data Engineer, Gen AI
Role: Data Engineer
We are seeking a skilled Data Engineer to support our organization's generative AI initiatives. In this role, you will be responsible for designing, implementing, and maintaining the data infrastructure and pipelines necessary to enable large-scale generative AI model deployment and operation.
Key Responsibilities:
Design and build scalable data pipelines to ingest, process, and store large volumes of training data for generative AI models Implement data preprocessing and feature engineering workflows using Spark and Delta Lake to prepare data for model training and inference.Develop and maintain data quality checks and monitoring systems to ensure data integrity.Design and implement efficient data storage solutions using Delta Lake and Databricks SQL warehouses to support AI workloads.Collaborate with ML engineers and data scientists to optimize data flows for model training and serving.Implement data governance and security best practices for sensitive training data.Optimize data infrastructure for high-performance AI model serving in production environments.Troubleshoot data-related issues in AI pipelines and implement solutionsRequirements:
Bachelor's or Master's degree in Computer Science, Data Science, or related field 3+ years of experience in data engineering rolesStrong programming skills in Python and SQL Experience with Databricks, Apache Spark, and Delta LakeFamiliarity with cloud platforms (AWS, GCP, or Azure) and their data services Knowledge of data modeling, ETL processes, and data pipeline architecturesExperience with version control systems (e.g. Git) and CI/CD practicesUnderstanding of data privacy and security considerationsPreferred Qualifications:
Experience supporting machine learning or AI projects in production environmentsFamiliarity with containerization and orchestration tools like Docker and KubernetesKnowledge of streaming data technologies like Kafka or KinesisExperience with MLOps practices and tools Understanding of large language models and generative AI architecturesSkills:
Experience with Delta Live Tables for building reliable, maintainable data pipelinesFamiliarity with Databricks SQL for querying and analyzing large datasetsKnowledge of Unity Catalog for data governance and access controlStrong problem-solving and analytical skillsExcellent communication and collaboration abilitiesAbility to work in a fast-paced, dynamic environment Passion for staying up to date with the latest developments in AI and data technologies
Role: Data Engineer
We are seeking a skilled Data Engineer to support our organization's generative AI initiatives. In this role, you will be responsible for designing, implementing, and maintaining the data infrastructure and pipelines necessary to enable large-scale generative AI model deployment and operation.
Key Responsibilities:
Design and build scalable data pipelines to ingest, process, and store large volumes of training data for generative AI models Implement data preprocessing and feature engineering workflows using Spark and Delta Lake to prepare data for model training and inference.Develop and maintain data quality checks and monitoring systems to ensure data integrity.Design and implement efficient data storage solutions using Delta Lake and Databricks SQL warehouses to support AI workloads.Collaborate with ML engineers and data scientists to optimize data flows for model training and serving.Implement data governance and security best practices for sensitive training data.Optimize data infrastructure for high-performance AI model serving in production environments.Troubleshoot data-related issues in AI pipelines and implement solutionsRequirements:
Bachelor's or Master's degree in Computer Science, Data Science, or related field 3+ years of experience in data engineering rolesStrong programming skills in Python and SQL Experience with Databricks, Apache Spark, and Delta LakeFamiliarity with cloud platforms (AWS, GCP, or Azure) and their data services Knowledge of data modeling, ETL processes, and data pipeline architecturesExperience with version control systems (e.g. Git) and CI/CD practicesUnderstanding of data privacy and security considerationsPreferred Qualifications:
Experience supporting machine learning or AI projects in production environmentsFamiliarity with containerization and orchestration tools like Docker and KubernetesKnowledge of streaming data technologies like Kafka or KinesisExperience with MLOps practices and tools Understanding of large language models and generative AI architecturesSkills:
Experience with Delta Live Tables for building reliable, maintainable data pipelinesFamiliarity with Databricks SQL for querying and analyzing large datasetsKnowledge of Unity Catalog for data governance and access controlStrong problem-solving and analytical skillsExcellent communication and collaboration abilitiesAbility to work in a fast-paced, dynamic environment Passion for staying up to date with the latest developments in AI and data technologies