Proliance Consulting
Sr. DataBricks Architect
Proliance Consulting - Washington, District of Columbia, us, 20022
Work at Proliance Consulting
Overview
- View job
Overview
We are seeking a talented and experienced Senior Data Architect to join our Cloud Center of Excellence. The candidate will work with our DataOps team to design, implement, and manage Azure Databricks clusters and workspaces within a centralized cloud environment. The ideal candidate must have a strong background in data engineering and infrastructure, with specific experience in related Azure services, including Azure Databricks. The position requires expertise in both architecture and the ability to perform hands-on engineering tasks at a senior level. Relevant certifications such as
Databricks Certified Professional Data Engineer,
Microsoft Certified: Azure Data Engineer Associate , and
Microsoft Certified: Azure Solutions Architect Expert are required for this position. Length:
Through EOY Associate Vendors:
We are accepting applications from candidates who are currently authorized to work in the US for any employer without sponsorship. Location:
Candidates must be located in PT or MT timezone Role & Responsibilities
Design, implement, and manage Azure Databricks clusters and workspaces within a centralized cloud environment using a Hub and Spoke Architecture. Experience with data mesh architecture using Azure Databricks, including creating decentralized data ownership, establishing domain-oriented data products, and ensuring data interoperability across various domains is also a plus. Configure and optimize Azure Databricks environments for performance, scalability, and security. Develop and maintain automated deployment scripts using Terraform for Databricks infrastructure. Implement best practices for Databricks cluster management, including autoscaling, cluster policies, and job scheduling. Troubleshoot and resolve issues related to Databricks clusters, including performance bottlenecks, connectivity problems, and configuration errors. Set up monitoring and alerting for Databricks clusters to ensure proactive identification and resolution of issues. Collaborate with data engineering teams to integrate Databricks with other Azure services (e.g., Azure Data Lake Storage). Set up cost management dashboards, alerts, and budgets to control Databricks costs and ensure efficient utilization. Train data teams on how to effectively use Databricks for their data processing and analytics needs. Develop and maintain IaC scripts using Terraform to automate the provisioning and management of cloud infrastructure. Implement infrastructure changes and updates through version-controlled IaC repositories. Ensure compliance with security and governance policies in IaC deployments. Collaborate with DevOps and cloud engineering teams to streamline infrastructure deployment processes. Conduct code reviews and ensure adherence to Terraform best practices. Troubleshoot and resolve issues related to Terraform deployments. Work with data teams to train them on incorporating Databricks best practices into their solutions & workflows. Create and maintain documentation for Databricks and DataOps best practices. Required Qualifications:
At least 5 years of proven experience in deploying and managing Azure Databricks clusters and workspaces. Expert-level proficiency in Azure Databricks is required for this position. Experience using Databricks asset bundles for efficient management and deployment of Databricks resources. Strong understanding of Databricks architecture, including cluster management, job scheduling, and workspace configuration. Experience with Databricks integration with Azure services (e.g., Azure Data Lake Storage & Data Lake Medallion Architecture). Extensive experience with IaC tools, particularly Terraform and ability to write and maintain Terraform scripts for cloud infrastructure deployment. Experience in troubleshooting and resolving Terraform deployment issues. Experience with CI/CD pipelines and automation tools such as Azure DevOps and GitHub. Bachelor's or Master's degree in Computer Science, Engineering, or a related field. Databricks Certified Professional Data Engineer, Microsoft Certified: Azure Data Engineer Associate, and Microsoft Certified: Azure Solutions Architect Expert are required for this position. Candidates must work Pacific Standard Time hours. East Coast & Central Time Zone candidates will not be considered. Desired Qualifications:
Developed, deployed, and optimized AI and machine learning solutions on the Databricks platform, utilizing frameworks such as Spark MLlib and TensorFlow to enhance data processing efficiency and predictive analytics performance. Certifications including Databricks Certified Generative AI Engineer Associate and/or Microsoft Azure AI Engineer Associate. Seniority level
Not Applicable Employment type
Contract Job function
Information Technology Industries
Staffing and Recruiting and IT Services and IT Consulting
#J-18808-Ljbffr
Databricks Certified Professional Data Engineer,
Microsoft Certified: Azure Data Engineer Associate , and
Microsoft Certified: Azure Solutions Architect Expert are required for this position. Length:
Through EOY Associate Vendors:
We are accepting applications from candidates who are currently authorized to work in the US for any employer without sponsorship. Location:
Candidates must be located in PT or MT timezone Role & Responsibilities
Design, implement, and manage Azure Databricks clusters and workspaces within a centralized cloud environment using a Hub and Spoke Architecture. Experience with data mesh architecture using Azure Databricks, including creating decentralized data ownership, establishing domain-oriented data products, and ensuring data interoperability across various domains is also a plus. Configure and optimize Azure Databricks environments for performance, scalability, and security. Develop and maintain automated deployment scripts using Terraform for Databricks infrastructure. Implement best practices for Databricks cluster management, including autoscaling, cluster policies, and job scheduling. Troubleshoot and resolve issues related to Databricks clusters, including performance bottlenecks, connectivity problems, and configuration errors. Set up monitoring and alerting for Databricks clusters to ensure proactive identification and resolution of issues. Collaborate with data engineering teams to integrate Databricks with other Azure services (e.g., Azure Data Lake Storage). Set up cost management dashboards, alerts, and budgets to control Databricks costs and ensure efficient utilization. Train data teams on how to effectively use Databricks for their data processing and analytics needs. Develop and maintain IaC scripts using Terraform to automate the provisioning and management of cloud infrastructure. Implement infrastructure changes and updates through version-controlled IaC repositories. Ensure compliance with security and governance policies in IaC deployments. Collaborate with DevOps and cloud engineering teams to streamline infrastructure deployment processes. Conduct code reviews and ensure adherence to Terraform best practices. Troubleshoot and resolve issues related to Terraform deployments. Work with data teams to train them on incorporating Databricks best practices into their solutions & workflows. Create and maintain documentation for Databricks and DataOps best practices. Required Qualifications:
At least 5 years of proven experience in deploying and managing Azure Databricks clusters and workspaces. Expert-level proficiency in Azure Databricks is required for this position. Experience using Databricks asset bundles for efficient management and deployment of Databricks resources. Strong understanding of Databricks architecture, including cluster management, job scheduling, and workspace configuration. Experience with Databricks integration with Azure services (e.g., Azure Data Lake Storage & Data Lake Medallion Architecture). Extensive experience with IaC tools, particularly Terraform and ability to write and maintain Terraform scripts for cloud infrastructure deployment. Experience in troubleshooting and resolving Terraform deployment issues. Experience with CI/CD pipelines and automation tools such as Azure DevOps and GitHub. Bachelor's or Master's degree in Computer Science, Engineering, or a related field. Databricks Certified Professional Data Engineer, Microsoft Certified: Azure Data Engineer Associate, and Microsoft Certified: Azure Solutions Architect Expert are required for this position. Candidates must work Pacific Standard Time hours. East Coast & Central Time Zone candidates will not be considered. Desired Qualifications:
Developed, deployed, and optimized AI and machine learning solutions on the Databricks platform, utilizing frameworks such as Spark MLlib and TensorFlow to enhance data processing efficiency and predictive analytics performance. Certifications including Databricks Certified Generative AI Engineer Associate and/or Microsoft Azure AI Engineer Associate. Seniority level
Not Applicable Employment type
Contract Job function
Information Technology Industries
Staffing and Recruiting and IT Services and IT Consulting
#J-18808-Ljbffr