Hire Talent
Lead Data Engineer
Hire Talent, Newark, New Jersey, us, 07175
Lead Data Engineer VMS0033161 PrudentialLevel 3 (potentially level 4)Right to hire (conversion is possible)Long term requirement (Initial couple months will want to join them FT)Narayana Murthy is HMInterview process: about 3 rounds of interviews
Notes:
Chief Data OfficeProvided data to entire CEO offices.EDP corporate teamAWS platform, Pyspark, HadoopUnderstanding technical requirements and give guidelines to the team.Help implement the development structure for the team.Acts as a bridge for development.2-3 members he will be leading.AWS Solution architect certified is helpful. (nice to have but very useful)Candidates need to understand why the solution is built and how does it solve the problem.Sourcing the problem, creating a solution, looking to the market to create a solution (industry standard/new tech), implement the solution and look at.Need to build end to end solutions with Pyspark, AWS glue, API Calls (Fullstack dev comes into play here)They already have a pre-built data platform.No coding at all, needs to know what needs to be done. Will work with the business and work as a bridge.Bridge for EDP corporate teamNeed to be able to explain the how they build pipeline and why they used such technologyStruggles
Candidates do not understand the business problem, what is the solution.Candidates have never done solutions themselves.
9p Manager (Data Engineering Lead, Data Team) - Engineering lead to help the design/build of the CDO enterprise data projects.ResponsibilitiesImplement and support
end-to-end data lake/warehousing/mart/business intelligence/ analytics/ services solutions (ingest, storage, integration, processing, services, access) in AWSdata lake data intake/request/onboarding services and service documentationdata lake ingestion services for batch/real time data ingest and service documentationdata processing services (ETL/ELT) for batch/real time (Glue/Kinesis/EMR) and service documentationdata storage services for data lake (S3)/ data warehouses (RDS/Redshift)/ data marts and service documentationdata services layer including Athena, Redshift, RDS, microservices and APIspipeline orchestration services including lambda, step functions, MWAA (optional)data security services (IAM/KMS/SM/encryption/anonymization/RBAC) and service documentationdata access provisioning services (Accounts, IAM Roles RBAC), processes, documentation and educationdata provisioning services for data consumption patterns including microservices, APIs and extractsmetadata capture and catalog services for data lake(S3/Athena), data warehouses (RDS/Redshift), Microservices/APIsmetadata capture and catalog services for pipeline/log data for monitoring /supportImplement CI/CD pipelinesPrepare documentation for data projects utilizing AWS based enterprise data platformImplement high velocity streaming solutions using Amazon Kinesis, SQS, and SMSMigrate data from traditional relational database systems to AWS relational databases such as Amazon RDS, Aurora, and RedshiftMigrate data from traditional file systems and NAS shares to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and RedshiftMigrate data from APIs to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and RedshiftProvide cost/spend monitoring reporting for AWS based data platform data initiativesProvide governance/audit reporting for access of AWS based data platformLead the implementation of a data lake strategy to enable LOBs and Corporate Functions with a robust, holistic view of data - driven decision makingServe as delivery lead for EDP data initiatives product ownerPartner with immediate engineering team, product owner, IT, partners on EDP agendaProvide technology thought leadership, consulting, and coaching/mentoringEstablish development, qa, stage and production migration/support processesEstablish best practices for development and support teamsDeliver end-end data initiatives from ingest-consume via microservices/ apis, jdbc/ odbc, file extracts etc.Work with scrum master to develop and own backlog, stories, epics, sprintsQualifications
Bachelor's degree in Computer Science, Software Engineering, MIS or equivalent combination of education and experienceExperience implementing, supporting data lakes, data warehouses and data applications on AWS for large enterprisesProgramming experience with Java, Python/ Scala, Shell scriptingSolid experience of AWS services such as CloudFormation, S3, Glue, EMR/ Spark, RDS, Redshift, DynamoDB, Lambda, Step Functions, IAM, KMS, SM etc.Solid experience implementing solutions on AWS based data lakesExperience implementing metadata solutions leveraging AWS non-relational data solutions such as ElastiCache and DynamoDBAWS Solutions Architect or AWS Big Data Certification preferredExperience in AWS data lake/data warehouse/business analyticsExperience and understanding of various core AWS services such as IAM, Cloud Formation, EC2, S3, EMR/Spark, Glue, Datasync, CloudHealth, CloudWatch, Lambda, Athena, and RedshiftExperience in system analysis, design, development, and implementation of data ingestion pipeline in AWSExperience with DevOps and Continuous Integration/ Delivery (CI/ CD) concepts and toolsExperience with business intelligence tools such as Tableau, Power BI or equivalentKnowledge of ETL/ ELTExperience in production support from Level 1 to Level 3Awareness of Data Management & Governance tools
All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender, identity, national origin, disability, or protected veteran status.
Notes:
Chief Data OfficeProvided data to entire CEO offices.EDP corporate teamAWS platform, Pyspark, HadoopUnderstanding technical requirements and give guidelines to the team.Help implement the development structure for the team.Acts as a bridge for development.2-3 members he will be leading.AWS Solution architect certified is helpful. (nice to have but very useful)Candidates need to understand why the solution is built and how does it solve the problem.Sourcing the problem, creating a solution, looking to the market to create a solution (industry standard/new tech), implement the solution and look at.Need to build end to end solutions with Pyspark, AWS glue, API Calls (Fullstack dev comes into play here)They already have a pre-built data platform.No coding at all, needs to know what needs to be done. Will work with the business and work as a bridge.Bridge for EDP corporate teamNeed to be able to explain the how they build pipeline and why they used such technologyStruggles
Candidates do not understand the business problem, what is the solution.Candidates have never done solutions themselves.
9p Manager (Data Engineering Lead, Data Team) - Engineering lead to help the design/build of the CDO enterprise data projects.ResponsibilitiesImplement and support
end-to-end data lake/warehousing/mart/business intelligence/ analytics/ services solutions (ingest, storage, integration, processing, services, access) in AWSdata lake data intake/request/onboarding services and service documentationdata lake ingestion services for batch/real time data ingest and service documentationdata processing services (ETL/ELT) for batch/real time (Glue/Kinesis/EMR) and service documentationdata storage services for data lake (S3)/ data warehouses (RDS/Redshift)/ data marts and service documentationdata services layer including Athena, Redshift, RDS, microservices and APIspipeline orchestration services including lambda, step functions, MWAA (optional)data security services (IAM/KMS/SM/encryption/anonymization/RBAC) and service documentationdata access provisioning services (Accounts, IAM Roles RBAC), processes, documentation and educationdata provisioning services for data consumption patterns including microservices, APIs and extractsmetadata capture and catalog services for data lake(S3/Athena), data warehouses (RDS/Redshift), Microservices/APIsmetadata capture and catalog services for pipeline/log data for monitoring /supportImplement CI/CD pipelinesPrepare documentation for data projects utilizing AWS based enterprise data platformImplement high velocity streaming solutions using Amazon Kinesis, SQS, and SMSMigrate data from traditional relational database systems to AWS relational databases such as Amazon RDS, Aurora, and RedshiftMigrate data from traditional file systems and NAS shares to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and RedshiftMigrate data from APIs to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and RedshiftProvide cost/spend monitoring reporting for AWS based data platform data initiativesProvide governance/audit reporting for access of AWS based data platformLead the implementation of a data lake strategy to enable LOBs and Corporate Functions with a robust, holistic view of data - driven decision makingServe as delivery lead for EDP data initiatives product ownerPartner with immediate engineering team, product owner, IT, partners on EDP agendaProvide technology thought leadership, consulting, and coaching/mentoringEstablish development, qa, stage and production migration/support processesEstablish best practices for development and support teamsDeliver end-end data initiatives from ingest-consume via microservices/ apis, jdbc/ odbc, file extracts etc.Work with scrum master to develop and own backlog, stories, epics, sprintsQualifications
Bachelor's degree in Computer Science, Software Engineering, MIS or equivalent combination of education and experienceExperience implementing, supporting data lakes, data warehouses and data applications on AWS for large enterprisesProgramming experience with Java, Python/ Scala, Shell scriptingSolid experience of AWS services such as CloudFormation, S3, Glue, EMR/ Spark, RDS, Redshift, DynamoDB, Lambda, Step Functions, IAM, KMS, SM etc.Solid experience implementing solutions on AWS based data lakesExperience implementing metadata solutions leveraging AWS non-relational data solutions such as ElastiCache and DynamoDBAWS Solutions Architect or AWS Big Data Certification preferredExperience in AWS data lake/data warehouse/business analyticsExperience and understanding of various core AWS services such as IAM, Cloud Formation, EC2, S3, EMR/Spark, Glue, Datasync, CloudHealth, CloudWatch, Lambda, Athena, and RedshiftExperience in system analysis, design, development, and implementation of data ingestion pipeline in AWSExperience with DevOps and Continuous Integration/ Delivery (CI/ CD) concepts and toolsExperience with business intelligence tools such as Tableau, Power BI or equivalentKnowledge of ETL/ ELTExperience in production support from Level 1 to Level 3Awareness of Data Management & Governance tools
All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender, identity, national origin, disability, or protected veteran status.