Lead Data Engineer – Azure Databricks & ADF

For more than 20 years, we have been working with organizations large and small to help solve business challenges through technology. We bring a unique combination of engineering and strategy to Make Data Work for organizations.Our clients range from the travel and leisure industry to publishing, retail and banking. The common thread between our clients is their commitment to making data work as seen through their investment in those efforts.In our quest to solve data challenges for our clients, we work with large enterprise, cloud based and marketing technology suites. We have a deep understanding of these solutions so we can help our clients make the most of their investment in an efficient way to have a data driven business.Softcrylic now joins forces with Hexaware to Make Data Work in bigger ways!Why Work at Softcrylic?

Softcrylic provides an engaging, team-focused, and rewarding work environment where people are excited about the work they do and passionate about delivering creative solutions to our clients.We are looking to add a

Lead Data Engineer – Azure Databricks & ADF

to our team! This is a Full-Time position and work location is in Philadelphia, PA.Job Summary:

We are looking for a highly skilled Technical Lead – Data Engineering with a strong background in building and optimizing data pipelines and transformation processes using Azure Databricks, Python, and Spark. The ideal candidate will have the ability to understand and convert Scala programs to PySpark within Databricks and possess hands-on experience with Azure Data Factory.Key Responsibilities:

Lead the design, development, and optimization of data pipelines and transformation processes using Azure Databricks and ADF.Understand and convert existing Scala programs to PySpark, ensuring seamless integration within Azure Databricks.Collaborate with cross-functional teams to gather and translate business requirements into technical solutions.Manage and utilize Azure Data Lake Gen 2 for efficient data storage and processing, ensuring data integrity and security.Implement and optimize data warehousing solutions using Synapse Analytics, focusing on dedicated SQL pools.Leverage Delta Lake to manage large-scale data workflows, ensuring high performance and reliability.Provide technical leadership and guidance to junior engineers, fostering a collaborative and innovative environment.Ensure data solutions align with best practices, compliance requirements, and industry standards, particularly in the healthcare domain, if applicable.Primary Skills:

Azure Databricks: Expertise in building and managing data pipelines using Azure Databricks, with a strong focus on leveraging Python and Spark.Data Transformation: Proven experience in developing data transformation processes within Azure Databricks.Scala to PySpark Conversion: Ability to understand and convert Scala programs to PySpark programs within Databricks.Azure Data Factory (ADF): Hands-on experience in designing and implementing ETL pipelines using Azure Data Factory.Secondary Skills:

Azure Data Lake Gen 2: Proficiency in managing and utilizing Azure Data Lake Gen 2 for data storage and processing.Synapse Analytics (Dedicated SQL Pool): Experience with Synapse Analytics, particularly with dedicated SQL pools for data warehousing and analytics.Delta Lake: Expertise in working with Delta Lake for managing and optimizing big data workflows.PySpark: Advanced skills in PySpark for data processing and transformation tasks.You have the talent. We have the opportunities. Apply for it!#J-18808-Ljbffr