Logo
Technogen International Company

Senior Data Engineer

Technogen International Company, Atlanta, GA

Senior Data Engineer

  • Design, construct, install, test, and maintain highly scalable data management systems on Azure and AWS, utilizing services such as Azure Data Lake, Azure Data Factory, AWS S3, and AWS Glue.
  • Integrate new data management technologies and software engineering tools into existing structures, focusing on Azure and AWS ecosystems.
  • Utilize Azure Databricks and AWS EMR for big data processing tasks to create data set processes for data modeling, mining, and production.
  • Develop analytics applications using Azure Synapse Analytics and AWS Redshift to leverage the data pipeline, providing actionable insights into key business performance metrics.
  • Collaborate with stakeholders across the Executive, Product, Data, and Design teams to support their data infrastructure needs, using Azure and AWS cloud services.
  • Implement data security and compliance procedures using Azure Security Center and AWS Identity and Access Management (IAM).
  • Evaluate and improve existing systems, incorporating new technologies and software engineering tools available in Azure and AWS ecosystems.
  • Monitor analytics and metrics results using tools like Azure Monitor and AWS CloudWatch.
  • Implement systems for tracking data quality and consistency using Azure and AWS services, ensuring data integrity across the pipeline.
  • Automate data workflows and pipeline processes using Azure Logic Apps and AWS Step Functions, integrating with Apache Airflow for orchestration.

Qualifications

  • Bachelor's or Master's degree in Computer Science, Engineering, or related field.
  • Proven experience as a Data Engineer or similar role, with a deep understanding of Azure and AWS cloud services relevant to data engineering (e.g., Azure Data Lake, Azure Data Factory, AWS S3, AWS Glue).
  • Expertise in SQL and programming languages, especially Python, for data manipulation and analysis.
  • Strong experience with big data tools such as Apache Spark and Hadoop.
  • Proficiency in data pipeline and workflow management tools, particularly Apache Airflow.
  • Experience with data modeling, data warehousing, and building ETL pipelines in both Azure (using Azure Data Factory, Azure Databricks) and AWS environments (using AWS Glue, AWS Data Pipeline).
  • Knowledge of stream-processing systems (e.g., Apache Kafka, Azure Stream Analytics, Amazon Kinesis) is a plus.
  • Familiarity with machine learning algorithms and data science principles is beneficial.
  • Strong problem-solving skills and ability to project manage and multitask.
  • Excellent communication and teamwork skills, with an ability to navigate and negotiate with stakeholders across the organization.