Senior Big Data Engineer

Job Description Senior Big Data Engineer Designing and implementing ETL Process using Bigdata Technologies (Pyspark, Spark, Hadoop, Hive, Sqoop, Oozie, Unix Shell Script) Experience in handling Query/code Optimization & query performance Managing the Data Lake environment like Compliance issues fix, Upgrades, Data Archival/Retrieval process to/from AWS S3 Developing Automation scripts to fix the Production issues and mentors Team to handle the Prod issues Managing & Setting up the Ingestion Developing data extraction and transformations to improve data reliability and quality Utilizing past experiences to identify and troubleshoot potential issues, ensuring smooth data operations Advising team members in analyzing relevant data and assessing alternatives Provide effective solutions to complex technical challenges, applying technical competence to understand and resolve issues. Develop high-performance ingestion pipelines to a Big Data platforms, ensuring consistent and reliable data flows