Python Pyspark Developer with Data Bricks

Job Summary We are seeking a Python PySpark Developer with expertise in Databricks to join our team. The ideal candidate will build and optimize data pipelines, collaborate with stakeholders to define and consolidate data assets, and migrate CMS data platforms into new environments. This role also involves creating scalable data processing libraries, implementing automation, and ensuring system performance and resilience on AWS cloud platforms. Key Responsibilities Collaborate with teams to design and implement features for the data platform. Build, maintain, and optimize data pipelines using PySpark and Databricks. Migrate and consolidate data platforms into cloud environments. Define and manage company data assets in collaboration with business analysts and solutions architects. Develop reusable libraries for data processing and maintain data governance frameworks. Build resilient, secure, and high-performing platforms in AWS using tools like EMR, Glue, S3, and DynamoDB. Implement automation processes and infrastructure as code using Terraform or CloudFormation. Conduct performance optimization and provide operational support for cloud platforms. Drive testing, chaos engineering exercises, and game-day simulations to improve system resilience. Collaborate with cross-functional teams to deliver innovative and scalable solutions. Required Qualifications 5 years of experience in data engineering roles. Proficiency in Python and SQL for data engineering tasks. Experience building data pipelines with Spark and working on Databricks. Hands-on experience with AWS cloud services (e.g., EC2, S3, DynamoDB, RDS). Strong analytical and problem-solving skills. Experience with relational and NoSQL datastores and cloud object stores. Advanced programming skills in Java, Python, or Go. Familiarity with DevOps tools such as Docker, Jenkins, and Kubernetes. Preferred Qualifications Experience with ACID-compliant formats such as Hudi, Iceberg, or Delta Lake. AWS Associate-level certification (Developer, Solutions Architect, or DevOps). Proficiency with data observability and monitoring tools like Datadog or Splunk. Knowledge of automation and continuous delivery methods using Jenkins, Maven, or Terraform. Familiarity with financial systems and mission-critical application development. Certifications AWS Developer, Solutions Architect, or DevOps Associate-level certification is preferred. Education: Bachelors Degree