Dexian DISYS
Data Engineer (Python/PySpark)
Dexian DISYS, Columbia, Maryland, United States, 21046
Data Engineer (Python/PySpark)
Job details
Posted23 August 2024
LocationColumbia, MD
Job typePermanent
Reference933458
Job description
Position: Data EngineerLocation: Columbia, MD (Hybrid)Job Type: C2H/Full-TimeHiring Manager Notes:Note:-
Candidates should be local to DMV. Will need to be in Columbia, MD office one day per week.This particular role will be HEAVY in Pyspark/Python language.Interview Process -
Initial screening with Manager (30 minutes); Team Interview (1 hour)Tech stack required:
PySpark, Python and SQLTech stack preferred:
DatabricksDay to day:
New project, specific to Python Dev (Intermediate to advanced). Previously everything was all batch processing, over last couple years they have real time streaming of clinical data now w/ CEND system.Things running 24/7 and shooting data directly to portal (50,000 users per month). RECENTLY created CEND (new alert system). New person will jump right into the service. Python and PySpark heavyIn office on Thursdays for now.Needs to have good comm skills to be able to speak to non-technical people. How did project they have done help the business/stake holders?Interviews will be technical as well as non-technical.
Job Description:Job Summary:The Data Engineer will be responsible for supporting HIT activities in the Baltimore/Washington region. The successful candidate will work to design, develop, and maintain a data analytics platform solution. Client is currently running an Azure Data Lake 2 / Azure Data Factory / Spark / Databricks infrastructure with approximately 300 production use cases that consolidate several streams of healthcare data. This role will primarily focus on the creation of new use cases through Databricks programming including communication, technical proficiency, collaboration, and willingness to learn new things. This role assists a senior data engineer to implement new ingestion pipelines and troubleshoot production issues.Essential Duties and Responsibilities:Include the following. Other duties may be assigned.Design, build, and maintain large, complex data processing pipelines that meet business requirementsEnsure the quality of deliverables by developing automated controls and performing unit, integration, and user acceptance testingDevelop scalable and re-usable frameworks for ingestion and transformation of large data setsTroubleshoot and perform root cause analysis on data pipeline issuesWork with source system owners and business owners to incorporate business changes into data pipelinesCreate comprehensive documentation of data workflows, processes, and infrastructure
Qualifications:To perform this job successfully, the incumbent must be able to perform each essential duty satisfactorily. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.Advanced programming skills in PySpark, Python, Spark SQLThorough understanding of Data Lakes, raw/enriched/curated layer concepts, and ETL within Azure frameworkProven development experience in building complex data pipelines for lakehouse/data warehouses using Agile methodologyExperience in architecture, design, and implementation using Databricks and Azure Data Factory
Required Experience/Education:4+ year college degree (required)4+ years of working with data integration (required)Experience working with Databricks (preferred)Dexian is a leading provider of staffing, IT, and workforce solutions with over 12,000 employees and 70 locations worldwide. As one of the largest IT staffing companies and the 2nd largest minority-owned staffing company in the U.S., Dexian was formed in 2023 through the merger of DISYS and Signature Consultants. Combining the best elements of its core companies, Dexian's platform connects talent, technology, and organizations to produce game-changing results that help everyone achieve their ambitions and goals.Dexian's brands include Dexian DISYS, Dexian Signature Consultants, Dexian Government Solutions, Dexian Talent Development and Dexian IT Solutions. Visit https://dexian.com/ to learn more.Dexian is an Equal Opportunity Employer that recruits and hires qualified candidates without regard to race, religion, sex, sexual orientation, gender identity, age, national origin, ancestry, citizenship, disability, or veteran status.
Job details
Posted23 August 2024
LocationColumbia, MD
Job typePermanent
Reference933458
Job description
Position: Data EngineerLocation: Columbia, MD (Hybrid)Job Type: C2H/Full-TimeHiring Manager Notes:Note:-
Candidates should be local to DMV. Will need to be in Columbia, MD office one day per week.This particular role will be HEAVY in Pyspark/Python language.Interview Process -
Initial screening with Manager (30 minutes); Team Interview (1 hour)Tech stack required:
PySpark, Python and SQLTech stack preferred:
DatabricksDay to day:
New project, specific to Python Dev (Intermediate to advanced). Previously everything was all batch processing, over last couple years they have real time streaming of clinical data now w/ CEND system.Things running 24/7 and shooting data directly to portal (50,000 users per month). RECENTLY created CEND (new alert system). New person will jump right into the service. Python and PySpark heavyIn office on Thursdays for now.Needs to have good comm skills to be able to speak to non-technical people. How did project they have done help the business/stake holders?Interviews will be technical as well as non-technical.
Job Description:Job Summary:The Data Engineer will be responsible for supporting HIT activities in the Baltimore/Washington region. The successful candidate will work to design, develop, and maintain a data analytics platform solution. Client is currently running an Azure Data Lake 2 / Azure Data Factory / Spark / Databricks infrastructure with approximately 300 production use cases that consolidate several streams of healthcare data. This role will primarily focus on the creation of new use cases through Databricks programming including communication, technical proficiency, collaboration, and willingness to learn new things. This role assists a senior data engineer to implement new ingestion pipelines and troubleshoot production issues.Essential Duties and Responsibilities:Include the following. Other duties may be assigned.Design, build, and maintain large, complex data processing pipelines that meet business requirementsEnsure the quality of deliverables by developing automated controls and performing unit, integration, and user acceptance testingDevelop scalable and re-usable frameworks for ingestion and transformation of large data setsTroubleshoot and perform root cause analysis on data pipeline issuesWork with source system owners and business owners to incorporate business changes into data pipelinesCreate comprehensive documentation of data workflows, processes, and infrastructure
Qualifications:To perform this job successfully, the incumbent must be able to perform each essential duty satisfactorily. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.Advanced programming skills in PySpark, Python, Spark SQLThorough understanding of Data Lakes, raw/enriched/curated layer concepts, and ETL within Azure frameworkProven development experience in building complex data pipelines for lakehouse/data warehouses using Agile methodologyExperience in architecture, design, and implementation using Databricks and Azure Data Factory
Required Experience/Education:4+ year college degree (required)4+ years of working with data integration (required)Experience working with Databricks (preferred)Dexian is a leading provider of staffing, IT, and workforce solutions with over 12,000 employees and 70 locations worldwide. As one of the largest IT staffing companies and the 2nd largest minority-owned staffing company in the U.S., Dexian was formed in 2023 through the merger of DISYS and Signature Consultants. Combining the best elements of its core companies, Dexian's platform connects talent, technology, and organizations to produce game-changing results that help everyone achieve their ambitions and goals.Dexian's brands include Dexian DISYS, Dexian Signature Consultants, Dexian Government Solutions, Dexian Talent Development and Dexian IT Solutions. Visit https://dexian.com/ to learn more.Dexian is an Equal Opportunity Employer that recruits and hires qualified candidates without regard to race, religion, sex, sexual orientation, gender identity, age, national origin, ancestry, citizenship, disability, or veteran status.