Orion Innovation
Sr Data Engineer - Pyspark / Databricks
Orion Innovation, Edison, New Jersey, us, 08818
Orion Innovation is a premier, award-winning, global business and technology services firm. Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity. We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.Sr Data Engineer - Pyspark / Databricks
Responsibilities
The Senior PySpark Developer should have at least 8 years of extensive experience in designing, building, and deploying Python and PySpark-based applications.Familiarity with Databricks and Delta Lake is essential. Experience in designing, developing, deploying, and testing within these platforms is crucial.Proficiency in ETL processes using Python, PySpark, and Azure data factory.Knowledge of Azure services such as ADLS, Synaptics, PowerBi and MS Fabric will be advantageous.Develop and maintain data processing pipelines using Spark in Databricks notebooks. Optimize data workflows for performance, scalability, and reliability. Experience in finetuning application performance required.Work closely with team of Analysts, pyspark developers / QA to understand data requirements and deliver actionable insights.Ensure compliance with data governance and security standards.Qualifications
Strong proficiency in Python and PySpark.A Senior PySpark Developer should have a solid understanding of system design principles and experience with data modelling.Experience with Apache Spark and big data processing.Effective verbal and written communication skills are essential for presenting detailed designs and collaborating with cross-functional teams.Combine technical expertise with problem-solving passion to deliver technical solutions for large-scale transformation projects.Skills Required
Data Lake/ ADLS Gen 2Azure DatabricksPySpark, PythonAzure data factory
#J-18808-Ljbffr
Responsibilities
The Senior PySpark Developer should have at least 8 years of extensive experience in designing, building, and deploying Python and PySpark-based applications.Familiarity with Databricks and Delta Lake is essential. Experience in designing, developing, deploying, and testing within these platforms is crucial.Proficiency in ETL processes using Python, PySpark, and Azure data factory.Knowledge of Azure services such as ADLS, Synaptics, PowerBi and MS Fabric will be advantageous.Develop and maintain data processing pipelines using Spark in Databricks notebooks. Optimize data workflows for performance, scalability, and reliability. Experience in finetuning application performance required.Work closely with team of Analysts, pyspark developers / QA to understand data requirements and deliver actionable insights.Ensure compliance with data governance and security standards.Qualifications
Strong proficiency in Python and PySpark.A Senior PySpark Developer should have a solid understanding of system design principles and experience with data modelling.Experience with Apache Spark and big data processing.Effective verbal and written communication skills are essential for presenting detailed designs and collaborating with cross-functional teams.Combine technical expertise with problem-solving passion to deliver technical solutions for large-scale transformation projects.Skills Required
Data Lake/ ADLS Gen 2Azure DatabricksPySpark, PythonAzure data factory
#J-18808-Ljbffr