Senior Data Ingestion Engineer

Senior Data Ingestion Engineer Most important for this role (key skills) Kafka (data streaming) Python/PySparkAWS Databricks Cloud Platform Unity Catalog Role Overview: This role will be responsible for conducting data ingestion and platform management on the Databricks platform. This role requires a deep understanding of: Data Lake ingestion processes and best practices Data Streaming (Kafka) ETL/ELT implementation CI/CD System integration tools Data pipeline management About the role: Ingest data from a variety of source systems and tailor ingestion approaches on a per-system basis Manage, maintain, and oversee ETL/ELT pipelines on the Databricks platform Optimize data pipelines for scalability and speed Document ingestion and integration flows and pipelines Use Airflow to schedule and automate ingestion jobs Manage metadata and master data in technical data catalog Ensure ELT/ETL design meets required security and compliance guidelines, and ensure PII management, flagging and risk assessment during ingestion Maintain ETL/ELT pipeline infrastructure and implement automated monitoring strategies Ensure adherence to SDLC best practices About you: Bring 5 years of experience in data engineering, ingestion pipelining, Data Streaming using Kafka and ETL/ELTHold a bachelor's degree in computer science, engineering, statistics, or related field Have hands-on experience with and understanding of the following: Kafka ( minimum 2 years)AWS (minimum 2 years)Python/PySpark (minimum 3 years)Spark/ScalaDatabricksUnity Catalog SQL ETL/ELT development, monitoring and pipelining using tools such as Apache Airflow Ingestion tools such as Dell Boomi Data quality guidelines CI/CD pipelines Agile Git and version control "PySpark" AND "kafka" AND "aws" AND "databricks" AND "data ingestion" AND "python" AND "unity catalog" AND "data streaming"