Beacon Hill
Data Engineer - Python
Beacon Hill, King Of Prussia, Pennsylvania, United States, 19406
To Apply for this Job Click HereLocation: Remote EST hoursDuration: Ongoing multi year contractRate: 50-58\/hrJob Summary: The Data Engineer is responsible for designing, developing, and maintaining data pipelines and infrastructure to support data-driven decision-making within the organization. This role requires strong proficiency in Python, SQL, cloud technologies, and Kubernetes to ensure the efficient processing, storage, and retrieval of data.Key Responsibilities:
Data Pipeline Development:
Design, build, and maintain scalable and reliable data pipelines using Python and SQL.Develop ETL (Extract, Transform, Load) processes to integrate data from various sources into data warehouses and databases.Ensure data quality and consistency across different data sources and systems.Cloud Data Infrastructure:
Implement and manage data storage and processing solutions on cloud platforms (e.g., AWS, Azure, Google Cloud).Utilize cloud data services such as BigQuery, Snowflake, Redshift, or similar to store and analyze large datasets.Configure and manage cloud resources for optimal performance and cost-efficiency.Containerization and Orchestration:
Deploy and manage data applications and services using Docker and Kubernetes.Develop and maintain Kubernetes manifests, Helm charts, and CI\/CD pipelines to automate data workflows.Monitor and troubleshoot containerized applications to ensure high availability and reliability.Performance Optimization:
Optimize data processing pipelines for performance and scalability.Tune SQL queries and data storage configurations to handle large volumes of data efficiently.Implement monitoring and logging solutions to track data pipeline performance and identify issues.Collaboration and Communication:
Work closely with data scientists, analysts, and other stakeholders to understand data requirements and provide support.Collaborate with DevOps and infrastructure teams to integrate data solutions with existing systems.Document data processes, workflows, and configurations for transparency and knowledge sharing.Qualifications:
Experience: Experience as a Data Engineer or in a similar role with a strong focus on Python, SQL, and cloud technologies.
Technical Skills:
Proficiency in Python for data engineering tasks, including scripting and automation.Advanced SQL skills for querying and manipulating data.Hands-on experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and related data services.Experience with containerization and orchestration technologies such as Docker and Kubernetes.
T1390869-PHI_1726235385To Apply for this Job Click Here
Data Pipeline Development:
Design, build, and maintain scalable and reliable data pipelines using Python and SQL.Develop ETL (Extract, Transform, Load) processes to integrate data from various sources into data warehouses and databases.Ensure data quality and consistency across different data sources and systems.Cloud Data Infrastructure:
Implement and manage data storage and processing solutions on cloud platforms (e.g., AWS, Azure, Google Cloud).Utilize cloud data services such as BigQuery, Snowflake, Redshift, or similar to store and analyze large datasets.Configure and manage cloud resources for optimal performance and cost-efficiency.Containerization and Orchestration:
Deploy and manage data applications and services using Docker and Kubernetes.Develop and maintain Kubernetes manifests, Helm charts, and CI\/CD pipelines to automate data workflows.Monitor and troubleshoot containerized applications to ensure high availability and reliability.Performance Optimization:
Optimize data processing pipelines for performance and scalability.Tune SQL queries and data storage configurations to handle large volumes of data efficiently.Implement monitoring and logging solutions to track data pipeline performance and identify issues.Collaboration and Communication:
Work closely with data scientists, analysts, and other stakeholders to understand data requirements and provide support.Collaborate with DevOps and infrastructure teams to integrate data solutions with existing systems.Document data processes, workflows, and configurations for transparency and knowledge sharing.Qualifications:
Experience: Experience as a Data Engineer or in a similar role with a strong focus on Python, SQL, and cloud technologies.
Technical Skills:
Proficiency in Python for data engineering tasks, including scripting and automation.Advanced SQL skills for querying and manipulating data.Hands-on experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and related data services.Experience with containerization and orchestration technologies such as Docker and Kubernetes.
T1390869-PHI_1726235385To Apply for this Job Click Here