VDart
Data Engineer (Azure Databricks)
VDart, Portland, Oregon, United States, 97204
Job Title : Data Engineer (Azure Databricks)
Job Type : Fulltime
Location : Pittsburgh, PA (Onsite/Hybrid or ready to travel) & Portland OR (Onsite/Hybrid or ready to travel)
Required Skills/Qualifications:7-15 years of relevant experienceBachelor's and/or master's degree in computer science or equivalent experience.Strong communication, analytical and problem-solving skills with a high attention to detail.Desired Experience:
At least two years of experience building and leading highly complex, technical engineering teams.Strong hands-on experience in DatabricksImplement scalable and sustainable data engineering solutions using tools such as Databricks, Azure, Apache Spark, and Python. The data pipelines must be created, maintained, and optimized as workloads move from development to production for specific use cases.Experience managing distributed teams preferred.Comfortable working with ambiguity and multiple stakeholders.Comfortable working cross functionality with product management and directly with customers; ability to deeply understand product and customer personas.Expertise on Azure Cloud platformGood SQL knowledgeKnowledge on orchestrating workloads on cloudAbility to set and lead the technical vision while balancing business driversStrong experience with PySpark, Python programmingProficiency with APIs, containerization and orchestration is a plusExperience handling large and complex sets of data from various sources and databasesSolid grasp of database engineering and design principles.Experience with Unity Catalog.Familiarity with CI/CD methods desiredGood to have Teradata Experience (not Mandatory)Responsibilities:
Manage end to end delivery by Investigating problem areas, working cross-functionally with product manager & other stakeholdersFollow the Agile development methodology; think strategically and execute methodicallyDevelop and manage capacity and growth projection forecasts of the environment within budgetsCreate and maintain optimal data pipeline architecture, Assemble large, complex data sets that meet functional / non-functional business requirementsDrive optimization, testing and tooling to improve quality of solutionsManage teams that build and operate high volume distributed systems in a SaaS environmentGreat at devising efficient processes that increase velocity and qualityTrain counterparts in the data pipelining and preparation techniques, which make it easier for them to integrate and consume the data they need for their own use cases.Promote the available data and analytics capabilities and expertise to business unit leaders and educate them in leveraging these capabilities in achieving their business goals.Accountable for operational effectiveness in performance, uptime, release management of the enterprise data platform.
Job Type : Fulltime
Location : Pittsburgh, PA (Onsite/Hybrid or ready to travel) & Portland OR (Onsite/Hybrid or ready to travel)
Required Skills/Qualifications:7-15 years of relevant experienceBachelor's and/or master's degree in computer science or equivalent experience.Strong communication, analytical and problem-solving skills with a high attention to detail.Desired Experience:
At least two years of experience building and leading highly complex, technical engineering teams.Strong hands-on experience in DatabricksImplement scalable and sustainable data engineering solutions using tools such as Databricks, Azure, Apache Spark, and Python. The data pipelines must be created, maintained, and optimized as workloads move from development to production for specific use cases.Experience managing distributed teams preferred.Comfortable working with ambiguity and multiple stakeholders.Comfortable working cross functionality with product management and directly with customers; ability to deeply understand product and customer personas.Expertise on Azure Cloud platformGood SQL knowledgeKnowledge on orchestrating workloads on cloudAbility to set and lead the technical vision while balancing business driversStrong experience with PySpark, Python programmingProficiency with APIs, containerization and orchestration is a plusExperience handling large and complex sets of data from various sources and databasesSolid grasp of database engineering and design principles.Experience with Unity Catalog.Familiarity with CI/CD methods desiredGood to have Teradata Experience (not Mandatory)Responsibilities:
Manage end to end delivery by Investigating problem areas, working cross-functionally with product manager & other stakeholdersFollow the Agile development methodology; think strategically and execute methodicallyDevelop and manage capacity and growth projection forecasts of the environment within budgetsCreate and maintain optimal data pipeline architecture, Assemble large, complex data sets that meet functional / non-functional business requirementsDrive optimization, testing and tooling to improve quality of solutionsManage teams that build and operate high volume distributed systems in a SaaS environmentGreat at devising efficient processes that increase velocity and qualityTrain counterparts in the data pipelining and preparation techniques, which make it easier for them to integrate and consume the data they need for their own use cases.Promote the available data and analytics capabilities and expertise to business unit leaders and educate them in leveraging these capabilities in achieving their business goals.Accountable for operational effectiveness in performance, uptime, release management of the enterprise data platform.