AWS Data Architect

Job Title: AWS Data Engineer

Location: Remote (San Diego, CA 92130)

Duration: 6 months (possibility of extension or conversion)

Job Description:

• As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and maintaining high-performance software system to manage analytical data pipelines that fuel the organization’s data strategy using software engineering best practices. Beyond technical expertise, you will also serve as a change leader, guiding teams through adopting new tools, technologies, and workflows to improve data management and processing.
• This position requires extensive hands-on data system design and coding experience, as well as the development of modern data pipelines (AWS Step functions, Prefect, Airflow, Luigi, Python, Spark, SQL) and associated code in AWS.
• You will work closely with stakeholders across the business to understand their data needs, ensure scalability, and foster a culture of innovation and learning within the data engineering team and beyond.

Key Responsibilities:

• Be responsible for the overall architecture of a specific module within a product (e.g., Data-ingestion, near-real-time-data-processor, etc.), perform design and assist implementation considering system characteristics to produce optimal performance, reliability and maintainability.
• Provide technical guidance to team members, ensuring they are working towards the product's architectural goals.
• Create and manage RFCs (Request for Comments) and ADRs (Architecture Decision Records), Design notes and technical documentation for your module, following the architecture governance processes.
• Lead a team of data engineers, providing mentorship, setting priorities, and ensuring alignment with business goals.
• Architect, design, and build scalable data pipelines for processing large volumes of structured and unstructured data from various sources.
• Collaborate with software engineers, architects, and product teams to design and implement systems that enable real-time and batch data processing at scale.
• Be the go-to person for PySpark-based solutions, ensuring optimal performance and reliability for distributed data processing.
• Ensure that data engineering systems adhere to the best data security, privacy, and governance practices in line with industry standards.
• Perform code reviews for the product, ensuring adherence to company coding standards and best practices.
• Develop and implement monitoring and alerting systems to ensure timely detection and resolution of data pipeline failures and performance bottlenecks.
• Act as a champion for new technologies, helping ease transitions and addressing concerns or resistance from team members.

Ideal Candidate:

• Experience leading a data engineering team with a strong focus on software engineering principles such as KISS, DRY, YAGNI etc.
• Must have experience in owning large, complex system architecture and hands-on experience designing and implementing data pipelines across large-scale systems.
• Experience implementing and optimizing data pipelines with AWS is a must.
• Production delivery experience in Cloud-based PaaS Big Data related technologies (EMR, Snowflake, Data bricks etc.)
• Experienced in multiple Cloud PaaS persistence technologies, and in-depth knowledge of cloud- based ETL offerings and orchestration technologies (AWS Step Function, Airflow etc.)
• Experienced in stream-based and batch processing, applying modern technologies
• Working experience with distributed file systems (S3, HDFC, ADLS), table formats (HUDI, Iceberg), and various open file formats (JSON, Parquet, Csv, etc.)
• Strong programming experience in PySpark, SQL, Python, etc.
• Database design skills including normalization/de-normalization and data warehouse design
• Knowledge and understanding of relevant legal and regulatory requirements, such as SOX, PCI, HIPAA, Data Protection
• Experience in the healthcare industry, a plus
• A collaborative and informative mentality is a must.

Toolset:

• AWS, preferably AWS certified Data Engineer and AWS certified Solutions Architect.
• Proficiency in at least one programming language C#, GoLang, JavaScript or ReactJs.
• Spark / Python / SQL.
• Snowflake/ Databricks / Synapse / MS SQL Server.
• ETL / Orchestration Tools (Step Function, DBT etc.).
• ML / Notebooks.

Education and experience required:

• Bachelors or master’s in computer science, Information Systems, or an engineering field or relevant experience.
• 10+ years of related experience in developing data solutions and data movement.