Azure Data Engineer with Databricks Expertise

Title : Azure Data Engineer with Databricks Expertise Location : NYC, NY/Iselin, NJ [Hybrid] Type : Long term contract Job Description: We are seeking highly skilled Azure Data Engineer with strong expertise in Databricks to join our data team. The ideal candidate will design, implement, and optimize large-scale data pipeline, ensuring scalability, reliability, and performance. This role involves working closely with multiple teams and business stakeholders to deliver cutting-edge data solutions. Key Responsibilities: Data Pipeline Development: Build and maintain scalable ETL/ELT pipelines using Databricks. Leverage PySpark/Spark and SQL to transform and process large datasets. Integrate data from multiple sources including Azure Blob Storage, ADLS and other relational/non-relational systems. Collaboration & Analysis: Work Closely with multiple teams to prepare data for dashboard and BI Tools. Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions. Performance & Optimization: Optimize Databricks workloads for cost efficiency and performance. Monitor and troubleshoot data pipelines to ensure reliability and accuracy. Governance & Security: Implement and manage data security, access controls and governance standards using Unity Catalog. Ensure compliance with organizational and regulatory data policies. Deployment: Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks, and configurations across environments. Manage version control for Databricks artifacts and collaborate with team to maintain development best practices.