EtiVenture, Inc
Senior Data Engineer
EtiVenture, Inc, San Antonio, Texas, United States, 78258
Identify and evaluate emerging technologies, tools, and trends that can drive innovation and improve the efficiency and effectiveness of our data engineering processes.
Design, develop, and own robust, scalable data processing and data integration pipelines using Python, Airflow, PySpark, SparkSQL, and REST API endpoints to ingest data from various external data sources to Data Lake.
Design and develop batch and streaming data pipelines using Apache Kafka, Flink etc. Design and develop dimensional data model techniques (snowflake and star schemas, dimension, fact tables, slowly changing dimensions, periodic snapshot tables etc.)
Design and build data models for optimal storage and retrieval.Collaborate with cross-functional teams, such as data scientists, software engineers, and business stakeholders to understand data requirements and deliver solutions that meet business needs.
Enforce data governance policies and practices to maintain data integrity, security, and compliance with relevant regulations.
Work with developers to implement software processes and tools such as Git, CI/CD pipelines, and orchestration tools like Airflow.REQUIREMENTS: Bachelor degree in Computer Science, Information Technology or related or foreign equivalent + 60 months of experience in the job offered.
Employer will accept an Associate's degree in Computer Science or foreign equivalent or three years of academic studies or foreign equivalent towards a Bachelor of Science Degree in Computer Science.
Experience in using Kafka, Hive and Sqoop tools are required.
Design, develop, and own robust, scalable data processing and data integration pipelines using Python, Airflow, PySpark, SparkSQL, and REST API endpoints to ingest data from various external data sources to Data Lake.
Design and develop batch and streaming data pipelines using Apache Kafka, Flink etc. Design and develop dimensional data model techniques (snowflake and star schemas, dimension, fact tables, slowly changing dimensions, periodic snapshot tables etc.)
Design and build data models for optimal storage and retrieval.Collaborate with cross-functional teams, such as data scientists, software engineers, and business stakeholders to understand data requirements and deliver solutions that meet business needs.
Enforce data governance policies and practices to maintain data integrity, security, and compliance with relevant regulations.
Work with developers to implement software processes and tools such as Git, CI/CD pipelines, and orchestration tools like Airflow.REQUIREMENTS: Bachelor degree in Computer Science, Information Technology or related or foreign equivalent + 60 months of experience in the job offered.
Employer will accept an Associate's degree in Computer Science or foreign equivalent or three years of academic studies or foreign equivalent towards a Bachelor of Science Degree in Computer Science.
Experience in using Kafka, Hive and Sqoop tools are required.