Expedite Technology Solutions LLC
Databricks ETL Developer
Expedite Technology Solutions LLC, Washington, District of Columbia, us, 20022
Role Description :
Our project to move a data warehouse from Oracle and SAP Data Services technologies to Databricks is in its early stages.This is an opportunity to become a key player in an important element of the agency's cloud migration.Responsibilities:
Work with product owners and internal and external teams to develop automated data pipelines from heterogeneous sources, including file and data validation, exception handling and reporting, and applying business rules in ETL to populate the data warehouse in alignment with agency standards and guidelines.Collaborate with data analysts, engineers, and stakeholders to understand data requirements and develop ETL solutions that meet business needs.Optimize ETL workflows for performance, scalability, and reliability, ensuring timely and accurate data delivery.Develop and maintain standards and documentation for ETL processes, including data mappings, transformations, and workflow diagrams.Monitor ETL jobs, troubleshoot issues, and implement enhancements to improve data quality and system efficiency.Ensure compliance with data governance policies and security standards throughout the ETL lifecycle.Preferred Qualifications:
Proven experience as an ETL Developer or Data Engineer, with a strong understanding of data warehousing concepts and ETL tools.Experience with cloud platforms (e.g., AWS) and big data technologies (e.g., Databricks, Spark)Experience with ETL tools such as Databricks, SAP Data Services including designing and deploying ETL workflows across a variety of data sources to include JSON and Parquet files.Proficiency in SQL and scripting languages (e.g., Python) for data manipulation and automation.Familiarity with relational databases (e.g., Oracle, SQL Server, PostgreSQL) and data modeling techniques.Strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions.
Our project to move a data warehouse from Oracle and SAP Data Services technologies to Databricks is in its early stages.This is an opportunity to become a key player in an important element of the agency's cloud migration.Responsibilities:
Work with product owners and internal and external teams to develop automated data pipelines from heterogeneous sources, including file and data validation, exception handling and reporting, and applying business rules in ETL to populate the data warehouse in alignment with agency standards and guidelines.Collaborate with data analysts, engineers, and stakeholders to understand data requirements and develop ETL solutions that meet business needs.Optimize ETL workflows for performance, scalability, and reliability, ensuring timely and accurate data delivery.Develop and maintain standards and documentation for ETL processes, including data mappings, transformations, and workflow diagrams.Monitor ETL jobs, troubleshoot issues, and implement enhancements to improve data quality and system efficiency.Ensure compliance with data governance policies and security standards throughout the ETL lifecycle.Preferred Qualifications:
Proven experience as an ETL Developer or Data Engineer, with a strong understanding of data warehousing concepts and ETL tools.Experience with cloud platforms (e.g., AWS) and big data technologies (e.g., Databricks, Spark)Experience with ETL tools such as Databricks, SAP Data Services including designing and deploying ETL workflows across a variety of data sources to include JSON and Parquet files.Proficiency in SQL and scripting languages (e.g., Python) for data manipulation and automation.Familiarity with relational databases (e.g., Oracle, SQL Server, PostgreSQL) and data modeling techniques.Strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions.