Data Engineer

Job ID : 1778 Posting Date : 02/01/2022 Job Description Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in data engineering development tools and technologies. Develop, test, and maintain optimal data processing pipelines and related architectures, ensuring the overall solution will support business requirements. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using cloud-native technologies. Work with a team of developers with deep experience in Hadoop, spark, hive, machine learning, distributed microservices, and full-stack system. Design and Develop data transformation scripts to Transform data using Azure Databricks and Python. Design and Develop complex SQL queries Azure SQL for QA testing and report/data validation. Model and build large, complex data sets that meet functional/non-functional business requirements. Develop and implement processes for data ingestion, extraction, mining, and production. Employ a variety of languages, tools, and techniques to marry systems and the data generated from those systems together to maximize the judicious value of the data. Manage data migrations/conversions and troubleshoot data processing issues. Utilize programming languages like Python Java, Scala, and RDBMS, and NoSQL databases and Cloud-based data warehousing services such as Snowflake. Perform unit tests and conduct reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance. Responsibilities Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in data engineering development tools and technologies. Develop, test, and maintain optimal data processing pipelines and related architectures, ensuring the overall solution will support business requirements. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using cloud-native technologies. Work with a team of developers with deep experience in Hadoop, spark, hive, machine learning, distributed microservices, and full-stack system. Design and Develop data transformation scripts to Transform data using Azure Databricks and Python. Design and Develop complex SQL queries Azure SQL for QA testing and report/data validation. Model and build large, complex data sets that meet functional/non-functional business requirements. Develop and implement processes for data ingestion, extraction, mining, and production. Employ a variety of languages, tools, and techniques to marry systems and the data generated from those systems together to maximize the judicious value of the data. Manage data migrations/conversions and troubleshoot data processing issues. Utilize programming languages like Python Java, Scala, and RDBMS, and NoSQL databases and Cloud-based data warehousing services such as Snowflake. Perform unit tests and conduct reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance. Education Level Bachelor's in Computer Science or Equivalent