InterSources
Senior Data Engineer
InterSources, Arizona City, Arizona, United States, 85123
Title: Senior Data EngineerLocation: Remote/TelecommuteDuration: Long term contract
Description: The primary responsibility of Senior Data Management Engineer is to build data pipelines, model andprepare data, perform complex data analysis to answer Business questions, build and automate data pipeline and quality framework to enable and promote self service data pipelines, assist in operationalizing the AI / Client Engineering solutions. This role is expected to lead and guide other team members and evangelize the design patterns as well as coding standards.This role plays an active part in our Data Modernization project to migrate the from on-prem platformssuch as IBM Netezza to cloud project
Responsibilities:
Team up with the engineering teams and enterprise architecture (EA) to define standards, design patterns, accelerators, development practices, DevOps and CI/CD automationCreate and maintain the data ingestion, quality testing and audit frameworkConduct complex data analysis to answer the queries from Business Users or Technology team partners either directly from Analysts or stemmed from one of the Reporting tools suchs PowerBI, Tableau, OBIEE.Build and automate the data ingestion, transformation and aggregation pipelines using Azure Data Factory, Databricks / Spark, Snowflake, Kafka as well as Enterprise Scheduler tools such as CA Workload automation or Control MSetup and evangelize the metadata driven approach to data pipelines to promote self serviceSetup and continuously improve the data quality and audit monitoring as well as alertingConstantly evaluate the process automation options and collaborate with engineering as well as architecture to review the proposed design.Demonstrate mastery of build and release engineering principles and methodologies including source control, branch management, build and smoke testing, archiving and retention practicesAdhere to and enhance and document the design principles, best practices by collaborating with Solution and in some cases Enterprise ArchitectsParticipate in and support the Data Academy and Data Literacy program to train the Business Users and Technology teams on DataRespond SLA driven production data quality or pipeline issuesWork in a fast-paced Agile/Scrum environmentIdentify and assist with implementation of DevOps practices in support of fully automated deploymentsDocument the Data Flow Diagrams, Data Models, Technical Data Mapping and Production Support Information for Data PipelinesFollow the Industry standard data security practices and evangelize the same across the team.
Requirements:
5+ years of experience in an Enterprise Data Management or Data Engineering role3+ of hands on experience in building metadata driven data pipelines using Azure Data Factory,Databricks / Spark for Cloud Datalake5+ years hands on experience with using one or more of the following for data analysis andwrangling Databricks, Python / PySpark, Jupyter NotebooksExpert level SQL knowledge on databases such as but not limited to Snowflake, Netezza, Oracle,Sql Server, MySQL, Teradata3+ years of hands on experience on one or more of big data technologies such as ClouderaHadoop, Pivotal, Vertica, MapR is a plusExperience working in a multi developer environment and hands on experience in using either azure devops or gitlabPreferably experienced in SLA driven Production Data Pipeline or Quality supportExperience or strong understanding of the traditional enterprise ETL platforms such as IBMDatastage, Informatica, Pentaho, Ab Initio etc.Functional knowledge of some of the following technologies - Terraform, Azure CLI, PowerShell, Containerization (Kubernetes, Docker)Functional knowledge of one or more Reporting tools such as PowerBI, Tableau, OBIEETeam player with excellent communication skills, ability to communicate with the customer directly and able to explain the status of the deliverables in scrum callsAbility to implement Agile methodologies and work in an Agile DevOps environment
Qualifications:
Bachelor's degree in Computer Science or Engineering or Mathematics or related field and 5+ years of experience in various cloud technologies within a large-scale organizationPersonal Attributes: Self-starter, Collaborative, Curious, Strong work ethic, highly motivated, Team orientedExperience designing and building complex data pipelines in an agile environmentExpertise on data analysis and wrangling using sql, python, databricksExperience with modern cloud development and design concepts; software development lifecycle; multi-developer code versioning and conflict resolution; planning, design, and problem resolution enterprise data applications / solutionsDemonstrated ability in developing a culture that embraces innovation, and challenges existing paradigms
About Us:InterSources Inc , a Certified Diverse Supplier, was founded in 2007 and offers innovative solutions to help clients with Digital Transformations across various domains and industries. Our history spans over 16 years and today we are an Award-Winning Global Software Consultancy solving complex problems with technology. We recognize that our employees and our clients are our strengths as the diverse talents and opportunities they bring to the table enable us to grow as a global platform and they are causally linked with our success. We provide strategic and technical advice, and we have expertise in areas covering
Artificial Intelligence, Cloud Migration, Custom Software Development, Data Analytics Infrastructure & Cloud Solutions, Cyber Security Services,
etc. We make reasonable accommodations for clients and employees and we do not discriminate based on any protected attribute including race, religion, color, national origin, gender sexual orientation, gender identity, age, or marital status. We also are a
Google Cloud partner
company. We align strategy with execution and provide secure service solutions by developing and using the latest technologies that thrive our resources to deliver industry-leading capabilities to our clients and customers, making it convenient for our clients to do business with InterSources Inc. Our teams also drive growth by refining technology-driven client experiences that put the users first, providing an unparalleled experience. This results in strengthening the core technologies of clients, enabling them to scale with flexibility, create seamless digital experiences and build lifelong relationships.
Description: The primary responsibility of Senior Data Management Engineer is to build data pipelines, model andprepare data, perform complex data analysis to answer Business questions, build and automate data pipeline and quality framework to enable and promote self service data pipelines, assist in operationalizing the AI / Client Engineering solutions. This role is expected to lead and guide other team members and evangelize the design patterns as well as coding standards.This role plays an active part in our Data Modernization project to migrate the from on-prem platformssuch as IBM Netezza to cloud project
Responsibilities:
Team up with the engineering teams and enterprise architecture (EA) to define standards, design patterns, accelerators, development practices, DevOps and CI/CD automationCreate and maintain the data ingestion, quality testing and audit frameworkConduct complex data analysis to answer the queries from Business Users or Technology team partners either directly from Analysts or stemmed from one of the Reporting tools suchs PowerBI, Tableau, OBIEE.Build and automate the data ingestion, transformation and aggregation pipelines using Azure Data Factory, Databricks / Spark, Snowflake, Kafka as well as Enterprise Scheduler tools such as CA Workload automation or Control MSetup and evangelize the metadata driven approach to data pipelines to promote self serviceSetup and continuously improve the data quality and audit monitoring as well as alertingConstantly evaluate the process automation options and collaborate with engineering as well as architecture to review the proposed design.Demonstrate mastery of build and release engineering principles and methodologies including source control, branch management, build and smoke testing, archiving and retention practicesAdhere to and enhance and document the design principles, best practices by collaborating with Solution and in some cases Enterprise ArchitectsParticipate in and support the Data Academy and Data Literacy program to train the Business Users and Technology teams on DataRespond SLA driven production data quality or pipeline issuesWork in a fast-paced Agile/Scrum environmentIdentify and assist with implementation of DevOps practices in support of fully automated deploymentsDocument the Data Flow Diagrams, Data Models, Technical Data Mapping and Production Support Information for Data PipelinesFollow the Industry standard data security practices and evangelize the same across the team.
Requirements:
5+ years of experience in an Enterprise Data Management or Data Engineering role3+ of hands on experience in building metadata driven data pipelines using Azure Data Factory,Databricks / Spark for Cloud Datalake5+ years hands on experience with using one or more of the following for data analysis andwrangling Databricks, Python / PySpark, Jupyter NotebooksExpert level SQL knowledge on databases such as but not limited to Snowflake, Netezza, Oracle,Sql Server, MySQL, Teradata3+ years of hands on experience on one or more of big data technologies such as ClouderaHadoop, Pivotal, Vertica, MapR is a plusExperience working in a multi developer environment and hands on experience in using either azure devops or gitlabPreferably experienced in SLA driven Production Data Pipeline or Quality supportExperience or strong understanding of the traditional enterprise ETL platforms such as IBMDatastage, Informatica, Pentaho, Ab Initio etc.Functional knowledge of some of the following technologies - Terraform, Azure CLI, PowerShell, Containerization (Kubernetes, Docker)Functional knowledge of one or more Reporting tools such as PowerBI, Tableau, OBIEETeam player with excellent communication skills, ability to communicate with the customer directly and able to explain the status of the deliverables in scrum callsAbility to implement Agile methodologies and work in an Agile DevOps environment
Qualifications:
Bachelor's degree in Computer Science or Engineering or Mathematics or related field and 5+ years of experience in various cloud technologies within a large-scale organizationPersonal Attributes: Self-starter, Collaborative, Curious, Strong work ethic, highly motivated, Team orientedExperience designing and building complex data pipelines in an agile environmentExpertise on data analysis and wrangling using sql, python, databricksExperience with modern cloud development and design concepts; software development lifecycle; multi-developer code versioning and conflict resolution; planning, design, and problem resolution enterprise data applications / solutionsDemonstrated ability in developing a culture that embraces innovation, and challenges existing paradigms
About Us:InterSources Inc , a Certified Diverse Supplier, was founded in 2007 and offers innovative solutions to help clients with Digital Transformations across various domains and industries. Our history spans over 16 years and today we are an Award-Winning Global Software Consultancy solving complex problems with technology. We recognize that our employees and our clients are our strengths as the diverse talents and opportunities they bring to the table enable us to grow as a global platform and they are causally linked with our success. We provide strategic and technical advice, and we have expertise in areas covering
Artificial Intelligence, Cloud Migration, Custom Software Development, Data Analytics Infrastructure & Cloud Solutions, Cyber Security Services,
etc. We make reasonable accommodations for clients and employees and we do not discriminate based on any protected attribute including race, religion, color, national origin, gender sexual orientation, gender identity, age, or marital status. We also are a
Google Cloud partner
company. We align strategy with execution and provide secure service solutions by developing and using the latest technologies that thrive our resources to deliver industry-leading capabilities to our clients and customers, making it convenient for our clients to do business with InterSources Inc. Our teams also drive growth by refining technology-driven client experiences that put the users first, providing an unparalleled experience. This results in strengthening the core technologies of clients, enabling them to scale with flexibility, create seamless digital experiences and build lifelong relationships.