Diversity Nexus
Senior Full Stack Data Engineer
Diversity Nexus, Princeton, New Jersey, us, 08543
REMOTE POSITIONProject DescriptionWe are looking for an experienced Full Stack Data Engineer to join our team for building a next-generation data platform built on Data Mesh architecture/principles. The ideal candidate should have extensive hands-on experience in building a big data platform, Big Data Technologies, Data Pipelines, backend development using Python, BI/Analytics tools as well as experience with DevOps, AWS, and UI Development in Angular JS. You will be responsible for designing, developing, and maintaining our web applications and data pipelines, as well as implementing CI/CD best practices.Responsibilities
Design, build, and maintain scalable and efficient data platforms using data engineering technologies such as Glue, EMR, Athena, Redshift, Lake Formation, Apache Spark, Hive, HDFS, and Trino.Build/manage data pipelines, and common data-related cross-cutting concerns like data catalog, data lineage, data quality, data profiling, data discovery, metadata management.Develop and maintain web applications using AngularJS and Python.Build/manage BI/Analytical dashboard reducing time to insight for the business stakeholders.Implement CI/CD pipelines using Terraform, Jenkins, Github actions, and Gitflow.Collaborate with cross-functional teams to develop and implement new features.Write clean, reusable, and efficient code.Participate in code reviews and ensure code quality.Develop and maintain APIs using Python and ensure API security and best practices are implemented.Implement SSO integration with Microsoft Azure AD using oAuth, OIDC, and SAML.Implement integration with AWS Cognito for user authentication and authorization.Ensure the application is optimized for maximum speed and scalability.Troubleshoot and debug issues as they arise.Implement DevOps best practices to ensure efficient application deployment and management.Collaborate with data scientists and analysts to integrate data analytics solutions with web applications.Stay up to date with emerging trends and technologies.Skills
Must have8+ years of experience in similar positions;Hands-on Experience with data engineering technologies such as AWS Glue, EMR, Athena, Redshift, Lake Formation, Apache Spark, Apache Hive, Apache Airflow, S3FS, Apache Hudi, and Trino.Extensive experience in building data pipelines using orchestration tools like Apache Airflow.Proven experience as a Full Stack Developer with AngularJS and Python.Strong understanding of web development technologies including HTML, CSS, and JavaScript.Experience working with RESTful APIs and JSON. Familiarity with microservices architecture.Experience with core AWS technologies such as EC2, ELB, Auto Scaling, S3, EFS, Lambda, API Gateway, Step Functions, Cloudwatch, VPC, Route 53, ACM.Hands-on experience with SQL and NoSQL databases.Hands-on experience with BI tools like Tableau, AWS QuickSight.Experience with Git or other version control systems.Understanding of agile development methodologies.Strong problem-solving skills.Excellent written and verbal communication skills.Ability to work independently and collaboratively in a team environment.Experience with cloud platforms such as AWS.Bachelor's degree in Computer Science, Engineering, or related field.Nice to haveAgile, Scrum framework 2+ years' experience on past projects.Knowledge of containerization technologies like Docker or Kubernetes.Experience with front-end frameworks like React or Vue.js.Experience with DevOps and CI/CD best practices.Experience with Terraform and the AWS provider.Experience with API development, security, and best practices.Experience with SSO integration with Microsoft Azure AD using oAuth, OIDC, and SAML.Experience with integration with AWS Cognito for user authentication and authorization.
#J-18808-Ljbffr
Design, build, and maintain scalable and efficient data platforms using data engineering technologies such as Glue, EMR, Athena, Redshift, Lake Formation, Apache Spark, Hive, HDFS, and Trino.Build/manage data pipelines, and common data-related cross-cutting concerns like data catalog, data lineage, data quality, data profiling, data discovery, metadata management.Develop and maintain web applications using AngularJS and Python.Build/manage BI/Analytical dashboard reducing time to insight for the business stakeholders.Implement CI/CD pipelines using Terraform, Jenkins, Github actions, and Gitflow.Collaborate with cross-functional teams to develop and implement new features.Write clean, reusable, and efficient code.Participate in code reviews and ensure code quality.Develop and maintain APIs using Python and ensure API security and best practices are implemented.Implement SSO integration with Microsoft Azure AD using oAuth, OIDC, and SAML.Implement integration with AWS Cognito for user authentication and authorization.Ensure the application is optimized for maximum speed and scalability.Troubleshoot and debug issues as they arise.Implement DevOps best practices to ensure efficient application deployment and management.Collaborate with data scientists and analysts to integrate data analytics solutions with web applications.Stay up to date with emerging trends and technologies.Skills
Must have8+ years of experience in similar positions;Hands-on Experience with data engineering technologies such as AWS Glue, EMR, Athena, Redshift, Lake Formation, Apache Spark, Apache Hive, Apache Airflow, S3FS, Apache Hudi, and Trino.Extensive experience in building data pipelines using orchestration tools like Apache Airflow.Proven experience as a Full Stack Developer with AngularJS and Python.Strong understanding of web development technologies including HTML, CSS, and JavaScript.Experience working with RESTful APIs and JSON. Familiarity with microservices architecture.Experience with core AWS technologies such as EC2, ELB, Auto Scaling, S3, EFS, Lambda, API Gateway, Step Functions, Cloudwatch, VPC, Route 53, ACM.Hands-on experience with SQL and NoSQL databases.Hands-on experience with BI tools like Tableau, AWS QuickSight.Experience with Git or other version control systems.Understanding of agile development methodologies.Strong problem-solving skills.Excellent written and verbal communication skills.Ability to work independently and collaboratively in a team environment.Experience with cloud platforms such as AWS.Bachelor's degree in Computer Science, Engineering, or related field.Nice to haveAgile, Scrum framework 2+ years' experience on past projects.Knowledge of containerization technologies like Docker or Kubernetes.Experience with front-end frameworks like React or Vue.js.Experience with DevOps and CI/CD best practices.Experience with Terraform and the AWS provider.Experience with API development, security, and best practices.Experience with SSO integration with Microsoft Azure AD using oAuth, OIDC, and SAML.Experience with integration with AWS Cognito for user authentication and authorization.
#J-18808-Ljbffr