Technogen International Company
Data Engineer
Technogen International Company, Dallas, TX
Data Engineer
Job Description
Data Pipeline Development:
Design, implement, and manage robust data pipelines using Python, PySpark, SQL to efficiently extract, transform, and load data from diverse sources(Batch & Streaming)
AWS Expertise
Demonstrate expertise in core AWS services such as AWS DMS, AWS Glue, AWS Step Functions, Amazon S3, Amazon Redshift, Amazon RDS, Amazon EMR, AWS IAM, AWS LAMBDA etc., and apply them to build scalable and reliable data solutions.
Data Modeling
Develop and maintain efficient data models to support the analytical and reporting needs.
Database Management
Administer databases using AWS services like Amazon RDS or Amazon Redshift, focusing on schema design, performance optimization, and monitoring.
Data Warehousing
Utilize Amazon Redshift or Amazon Snowflake to create high-performing analytical databases that empower data-driven decision-making.
ETL Best Practices
Implement industry best practices for ETL processes, including data validation, error handling, and data quality checks.
Performance Optimization
Optimize query performance through continuous tuning of databases and leveraging AWS's scalability capabilities.
Monitoring And Logging
Establish robust monitoring and logging mechanisms using AWS CloudWatch, Amazon CloudTrail, or comparable tools to ensure pipeline reliability.
Security And Compliance
Ensure adherence to security best practices and relevant compliance standards, tailoring solutions to meet GDPR, HIPAA, or other regulatory requirements.
Automation
Drive automation of deployment and scaling of data pipelines using infrastructure as code (IaC) tools like AWS CloudFormation and Terraform.
Collaboration
Collaborate closely with cross-functional teams, including data scientists, analysts, and other stakeholders, to understand their data needs and provide effective solutions.
Continuous Learning
Stay updated on the latest developments in AWS services and data engineering methodologies, applying new insights to enhance our data infrastructure.
Soft Skills
Exhibit strong communication skills to facilitate effective teamwork and interaction with diverse groups.
Job Description
Data Pipeline Development:
Design, implement, and manage robust data pipelines using Python, PySpark, SQL to efficiently extract, transform, and load data from diverse sources(Batch & Streaming)
AWS Expertise
Demonstrate expertise in core AWS services such as AWS DMS, AWS Glue, AWS Step Functions, Amazon S3, Amazon Redshift, Amazon RDS, Amazon EMR, AWS IAM, AWS LAMBDA etc., and apply them to build scalable and reliable data solutions.
Data Modeling
Develop and maintain efficient data models to support the analytical and reporting needs.
Database Management
Administer databases using AWS services like Amazon RDS or Amazon Redshift, focusing on schema design, performance optimization, and monitoring.
Data Warehousing
Utilize Amazon Redshift or Amazon Snowflake to create high-performing analytical databases that empower data-driven decision-making.
ETL Best Practices
Implement industry best practices for ETL processes, including data validation, error handling, and data quality checks.
Performance Optimization
Optimize query performance through continuous tuning of databases and leveraging AWS's scalability capabilities.
Monitoring And Logging
Establish robust monitoring and logging mechanisms using AWS CloudWatch, Amazon CloudTrail, or comparable tools to ensure pipeline reliability.
Security And Compliance
Ensure adherence to security best practices and relevant compliance standards, tailoring solutions to meet GDPR, HIPAA, or other regulatory requirements.
Automation
Drive automation of deployment and scaling of data pipelines using infrastructure as code (IaC) tools like AWS CloudFormation and Terraform.
Collaboration
Collaborate closely with cross-functional teams, including data scientists, analysts, and other stakeholders, to understand their data needs and provide effective solutions.
Continuous Learning
Stay updated on the latest developments in AWS services and data engineering methodologies, applying new insights to enhance our data infrastructure.
Soft Skills
Exhibit strong communication skills to facilitate effective teamwork and interaction with diverse groups.