Senior Data Engineer

Job Summary:

We are seeking a highly skilled Senior Data Engineer to join our dynamic technology team. The ideal candidate will have a strong background in data engineering, data architecture, and cloud-based data solutions, particularly with AWS services. This role is critical for building and maintaining robust data pipelines, ensuring data quality, and enabling data-driven decision-making across the organization.

Key Responsibilities:

• Data Pipeline Development: Design, implement, and maintain scalable data pipelines to process large volumes of data using AWS services, ensuring high availability and performance.
• Data Processing: Utilize Apache Spark, Databricks, or Snowflake to transform and analyze data from various sources, supporting business analytics and reporting needs.
• Data Modeling: Use DBT for data transformation, ensuring that data models are accurate, efficient, and maintainable.
• Collaboration: Work closely with data analysts, data scientists, and business stakeholders to understand data needs and deliver data solutions that meet those requirements.
• Data Quality Assurance: Monitor data quality metrics, identify discrepancies, and implement corrective measures to maintain data integrity and accuracy.
• Documentation: Create and maintain detailed documentation of data architectures, pipelines, and workflows for future reference and knowledge transfer.
• Optimization: Continuously evaluate and optimize data storage, processing, and retrieval strategies for efficiency and scalability.
• Troubleshooting: Investigate and resolve data issues in a timely manner, providing support for data-related queries and challenges.
• Best Practices: Promote best practices in data engineering, including coding standards, testing methodologies, and data governance.

Must-Have Skills:

• AWS Services: Strong experience with AWS data services (e.g., S3, Redshift, Glue, Lambda) for data storage, processing, and orchestration.
• SQL: Proficient in writing complex SQL queries for data extraction, manipulation, and analysis.
• Python: Solid programming skills in Python for data processing, automation, and API integration.
• Apache Spark: Hands-on experience with Apache Spark for distributed data processing and analytics.
• Data Lakes: Familiarity with Databricks or Snowflake as data warehousing solutions to support analytical workloads.
• DBT: Proficiency in using DBT for data transformation and building modular, testable data models.

Soft Skills:

• Excellent Communication: Ability to convey complex technical concepts to non-technical stakeholders and work collaboratively within cross-functional teams.
• Adaptability: Comfortable working in a fast-paced, evolving environment and adapting to changing priorities and technologies.
• Collaboration: Strong teamwork and interpersonal skills to foster a collaborative work environment.
• Analytical Skills: Exceptional analytical and problem-solving skills with a focus on detail, consistency, and quality.
• Critical Thinking: Ability to think critically and strategically about data architecture and engineering challenges.

Nice-to-Have Skills:

• Kafka: Familiarity with Apache Kafka for real-time data streaming and processing.
• CI/CD: Experience with Continuous Integration and Continuous Deployment tools and practices to automate the deployment of data solutions.
• Airflow: Knowledge of Apache Airflow for orchestrating and scheduling data workflows and pipelines.
• Fivetran: Understanding of Fivetran for seamless data integration from various sources to data warehouses.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• [Specify years of experience, e.g., 10+ Years in data engineering, data architecture, or a related field.
• Experience working with cloud-based data solutions and big data technologies.