NYU Langone Health
Lead Data Engineer - Azure & Databricks
NYU Langone Health, New York, New York, us, 10261
NYU Langone Health
is a world-class, patient-centered, integrated academic medical center, known for its excellence in clinical care, research, and education. It comprises more than 200 locations throughout the New York area, including five inpatient locations, a children's hospital, three emergency rooms, and a level 1 trauma center. Also part of NYU Langone Health is the Laura and Isaac Perlmutter Cancer Center, a National Cancer Institute designated comprehensive cancer center, and NYU Grossman School of Medicine, which since 1841 has trained thousands of physicians and scientists who have helped to shape the course of medical history. At NYU Langone Health, equity, diversity, and inclusion are fundamental values. We strive to be a place where our exceptionally talented faculty, staff, and students of all identities can thrive. We embrace diversity, inclusion, and individual skills, ideas, and knowledge.
Position Summary:We have an exciting opportunity to join our team as a Lead Data Engineer. In this role, the Enterprise Data and Analytics (EDA) department at NYU Langone Health plays a crucial role in modern healthcare organizations by leveraging data to enhance decision making, optimize operations, and improve patient outcomes. As a Senior Data Engineer within this department, you will play a critical role in designing and developing robust ETL pipelines to integrate diverse data sources. This role requires expertise in creating efficient data pipelines, handling both streaming and batch data processing, and ensuring data integrity throughout the ETL lifecycle. The engineer will implement monitoring solutions, optimize ETL jobs for performance, and provide comprehensive support from data ingestion to final output. With in-depth knowledge of the Databricks platform and strong analytical skills, this position will significantly enhance the department's ability to deliver high-quality data-driven insights and solutions.
Job Responsibilities:
Strategize, design, develop, and work with a team of dynamic and passionate data engineers to deliver automated cloud infrastructure and DevOps solutions.
Design and develop ETL code to integrate various data sources.
Create and maintain efficient data pipelines on the Databricks platform.
Develop, maintain, and optimize ETL pipelines for both streaming and batch data processing.
Ensure data integrity and consistency throughout the ETL lifecycle.
Provide comprehensive support for ETL processes from data ingestion to final output.
Write and execute unit tests and integration test cases for ETL code to ensure high-quality outcomes.
Implement monitoring solutions for ETL jobs to ensure timely and successful data processing.
Proactively identify, troubleshoot, and resolve issues in ETL workflows.
Optimize ETL jobs for maximum performance and efficiency through performance tuning and troubleshooting.
Mentor other data engineers in the team, cross-train, and provide guidance.
Minimum Qualifications:To qualify, you must have a minimum of a Bachelor’s degree in Computer Science, Information Systems, Engineering, or Data Science, and a minimum of 10 years of experience in designing, developing, and optimizing ETL processes. You should have 5 years of experience in developing/supporting a data platform in Azure Databricks, proficiency in creating and maintaining efficient data pipelines on the Databricks platform, and be well-rounded in working in a DevOps environment supporting processes in data platform in supporting business units and their use cases. Strong verbal and written communication skills with the ability to explain complex technical concepts to non-technical stakeholders, the ability to troubleshoot and resolve issues in a timely manner, and experience working in an Agile/Scrum environment are also required. You should be a team player with the ability to work collaboratively in a cross-functional team, possess strong analytical and problem-solving skills, and have the ability to work independently, handle multiple tasks simultaneously, and adapt quickly to change with a variety of people and work styles. A willingness to learn new technologies and continuously improve skills is essential.
Preferred Qualifications:In-depth knowledge of Databricks platform and technologies including Delta Lake, Databricks SQL, and Databricks Workflows. Experience with Azure cloud platforms and Azure Data Lake cloud storage. Knowledge of data warehousing, data modeling, and best practices. Proficiency in programming languages such as Python, SQL, Scala, or R. Experience with big data technologies such as Apache Spark, Hadoop, or Kafka. Knowledge of working with Hadoop for migration projects from Hadoop to Databricks. Familiarity with DevOps practices and tools such as CI/CD, Git, etc. Knowledge of infrastructure as Code (IaC) tools like Terraform. Experience with implementing data governance and security measures in a cloud environment.
Qualified candidates must be able to effectively communicate with all levels of the organization.
NYU Langone Health provides its staff with far more than just a place to work. Rather, we are an institution you can be proud of, an institution where you'll feel good about devoting your time and your talents.
NYU Langone Health is an equal opportunity and affirmative action employer committed to diversity and inclusion in all aspects of recruiting and employment. All qualified individuals are encouraged to apply and will receive consideration without regard to race, color, gender, gender identity or expression, sex, sexual orientation, transgender status, gender dysphoria, national origin, age, religion, disability, military and veteran status, marital or parental status, citizenship status, genetic information, or any other factor which cannot lawfully be used as a basis for an employment decision. We require applications to be completed online.
If you wish to view NYU Langone Health's EEO policies, please click here. Please click here to view the Federal "EEO is the law" poster or visit
this link
for more information.
NYU Langone Health provides a salary range to comply with the New York state Law on Salary Transparency in Job Advertisements. The salary range for the role is $81,325.15 - $135,541.73 Annually. Actual salaries depend on a variety of factors, including experience, specialty, education, and hospital need. The salary range or contractual rate listed does not include bonuses/incentives, differential pay, or other forms of compensation or benefits.
#J-18808-Ljbffr
is a world-class, patient-centered, integrated academic medical center, known for its excellence in clinical care, research, and education. It comprises more than 200 locations throughout the New York area, including five inpatient locations, a children's hospital, three emergency rooms, and a level 1 trauma center. Also part of NYU Langone Health is the Laura and Isaac Perlmutter Cancer Center, a National Cancer Institute designated comprehensive cancer center, and NYU Grossman School of Medicine, which since 1841 has trained thousands of physicians and scientists who have helped to shape the course of medical history. At NYU Langone Health, equity, diversity, and inclusion are fundamental values. We strive to be a place where our exceptionally talented faculty, staff, and students of all identities can thrive. We embrace diversity, inclusion, and individual skills, ideas, and knowledge.
Position Summary:We have an exciting opportunity to join our team as a Lead Data Engineer. In this role, the Enterprise Data and Analytics (EDA) department at NYU Langone Health plays a crucial role in modern healthcare organizations by leveraging data to enhance decision making, optimize operations, and improve patient outcomes. As a Senior Data Engineer within this department, you will play a critical role in designing and developing robust ETL pipelines to integrate diverse data sources. This role requires expertise in creating efficient data pipelines, handling both streaming and batch data processing, and ensuring data integrity throughout the ETL lifecycle. The engineer will implement monitoring solutions, optimize ETL jobs for performance, and provide comprehensive support from data ingestion to final output. With in-depth knowledge of the Databricks platform and strong analytical skills, this position will significantly enhance the department's ability to deliver high-quality data-driven insights and solutions.
Job Responsibilities:
Strategize, design, develop, and work with a team of dynamic and passionate data engineers to deliver automated cloud infrastructure and DevOps solutions.
Design and develop ETL code to integrate various data sources.
Create and maintain efficient data pipelines on the Databricks platform.
Develop, maintain, and optimize ETL pipelines for both streaming and batch data processing.
Ensure data integrity and consistency throughout the ETL lifecycle.
Provide comprehensive support for ETL processes from data ingestion to final output.
Write and execute unit tests and integration test cases for ETL code to ensure high-quality outcomes.
Implement monitoring solutions for ETL jobs to ensure timely and successful data processing.
Proactively identify, troubleshoot, and resolve issues in ETL workflows.
Optimize ETL jobs for maximum performance and efficiency through performance tuning and troubleshooting.
Mentor other data engineers in the team, cross-train, and provide guidance.
Minimum Qualifications:To qualify, you must have a minimum of a Bachelor’s degree in Computer Science, Information Systems, Engineering, or Data Science, and a minimum of 10 years of experience in designing, developing, and optimizing ETL processes. You should have 5 years of experience in developing/supporting a data platform in Azure Databricks, proficiency in creating and maintaining efficient data pipelines on the Databricks platform, and be well-rounded in working in a DevOps environment supporting processes in data platform in supporting business units and their use cases. Strong verbal and written communication skills with the ability to explain complex technical concepts to non-technical stakeholders, the ability to troubleshoot and resolve issues in a timely manner, and experience working in an Agile/Scrum environment are also required. You should be a team player with the ability to work collaboratively in a cross-functional team, possess strong analytical and problem-solving skills, and have the ability to work independently, handle multiple tasks simultaneously, and adapt quickly to change with a variety of people and work styles. A willingness to learn new technologies and continuously improve skills is essential.
Preferred Qualifications:In-depth knowledge of Databricks platform and technologies including Delta Lake, Databricks SQL, and Databricks Workflows. Experience with Azure cloud platforms and Azure Data Lake cloud storage. Knowledge of data warehousing, data modeling, and best practices. Proficiency in programming languages such as Python, SQL, Scala, or R. Experience with big data technologies such as Apache Spark, Hadoop, or Kafka. Knowledge of working with Hadoop for migration projects from Hadoop to Databricks. Familiarity with DevOps practices and tools such as CI/CD, Git, etc. Knowledge of infrastructure as Code (IaC) tools like Terraform. Experience with implementing data governance and security measures in a cloud environment.
Qualified candidates must be able to effectively communicate with all levels of the organization.
NYU Langone Health provides its staff with far more than just a place to work. Rather, we are an institution you can be proud of, an institution where you'll feel good about devoting your time and your talents.
NYU Langone Health is an equal opportunity and affirmative action employer committed to diversity and inclusion in all aspects of recruiting and employment. All qualified individuals are encouraged to apply and will receive consideration without regard to race, color, gender, gender identity or expression, sex, sexual orientation, transgender status, gender dysphoria, national origin, age, religion, disability, military and veteran status, marital or parental status, citizenship status, genetic information, or any other factor which cannot lawfully be used as a basis for an employment decision. We require applications to be completed online.
If you wish to view NYU Langone Health's EEO policies, please click here. Please click here to view the Federal "EEO is the law" poster or visit
this link
for more information.
NYU Langone Health provides a salary range to comply with the New York state Law on Salary Transparency in Job Advertisements. The salary range for the role is $81,325.15 - $135,541.73 Annually. Actual salaries depend on a variety of factors, including experience, specialty, education, and hospital need. The salary range or contractual rate listed does not include bonuses/incentives, differential pay, or other forms of compensation or benefits.
#J-18808-Ljbffr