ION GROUP
Senior Data Engineer - Data Mesh, New York
ION GROUP, New York, New York, us, 10261
We are seeking an experienced Data Engineer with AWS experience to join our newly created Service Consultant Sub-Practice and exclusively work with a mature and impactful long-term client in the financial services industry. This role requires a balance of independent work and teamwork, focusing on advancing the client's agenda. The ideal candidate will possess some understanding of the financial services sector, coupled with a proven track record of independent work and team collaboration. Join us to make a meaningful impact and contribute within our innovative team.
What you'll do:
Profiling of source system data for target state schema/table designDelivery of required datasets from the source systems into curated S3 buckets or folders, processed automatically via Databricks following medallion architectureEnsure data validations, normalization of datasets as per each business area's requirementsSupport testing and fine-tuning performance of workflows in DatabricksManage User Group identification per use case requirements (Creation of Okta groups, roles, permissions)Manage environment deployment process via CI/CDRequirements:
5+ years' experience as a Data Engineer5+ years' experience with Python and JavaStrong knowledge of SQLHands-on experience and understanding of API integrationsGood understanding of the Databricks platform and the data lakehouse / medallion architectureHands-on experience with Starburst / Trino and/ or AWS AthenaExperience with AWS-native technologies like Lambda, Glue and S3Database migrations experience with tools like Flyway, Liquibase. Atlas or equivalentCI / CD experience with Gitlab, GitHub or equivalentNice to haves:
Collibra & AWS integrationCollibra & Databricks integrationIaC experience with Terraform or CloudFormation
Lab49 is a specialist strategy, design, and technology consulting company dedicated to the financial services industry. We were founded in 2002 to fill a need in the Financial Technology marketplace for high-end execution and reliable delivery of technology projects.
We have office hubs in Midtown Manhattan, Washington DC, London, Budapest, New Delhi and Sydney.
Our Hybrid Work Model
At Lab49, we embrace a hybrid work model, offering you the flexibility to work from the office or remotely. We expect our Lab49ers to average 2-3 days per week in person and this could be at our midtown office in NYC or at a client site. We believe that both the flexibility and in person connection to each other and our clients build a collaborative culture and the opportunity to accelerate growth and innovation.
Lab49/ION is committed to maintaining a supportive and inclusive environment for people with diverse backgrounds and experiences. We respect the varied identities, abilities, cultures, and traditions of the individuals who comprise our organization and recognize the value that different backgrounds and points of view bring to our business.
Lab49/ION adheres to an equal employment opportunity policy that prohibits discriminatory practices or harassment against applicants or employees based on any legally impermissible factor.
What you'll do:
Profiling of source system data for target state schema/table designDelivery of required datasets from the source systems into curated S3 buckets or folders, processed automatically via Databricks following medallion architectureEnsure data validations, normalization of datasets as per each business area's requirementsSupport testing and fine-tuning performance of workflows in DatabricksManage User Group identification per use case requirements (Creation of Okta groups, roles, permissions)Manage environment deployment process via CI/CDRequirements:
5+ years' experience as a Data Engineer5+ years' experience with Python and JavaStrong knowledge of SQLHands-on experience and understanding of API integrationsGood understanding of the Databricks platform and the data lakehouse / medallion architectureHands-on experience with Starburst / Trino and/ or AWS AthenaExperience with AWS-native technologies like Lambda, Glue and S3Database migrations experience with tools like Flyway, Liquibase. Atlas or equivalentCI / CD experience with Gitlab, GitHub or equivalentNice to haves:
Collibra & AWS integrationCollibra & Databricks integrationIaC experience with Terraform or CloudFormation
Lab49 is a specialist strategy, design, and technology consulting company dedicated to the financial services industry. We were founded in 2002 to fill a need in the Financial Technology marketplace for high-end execution and reliable delivery of technology projects.
We have office hubs in Midtown Manhattan, Washington DC, London, Budapest, New Delhi and Sydney.
Our Hybrid Work Model
At Lab49, we embrace a hybrid work model, offering you the flexibility to work from the office or remotely. We expect our Lab49ers to average 2-3 days per week in person and this could be at our midtown office in NYC or at a client site. We believe that both the flexibility and in person connection to each other and our clients build a collaborative culture and the opportunity to accelerate growth and innovation.
Lab49/ION is committed to maintaining a supportive and inclusive environment for people with diverse backgrounds and experiences. We respect the varied identities, abilities, cultures, and traditions of the individuals who comprise our organization and recognize the value that different backgrounds and points of view bring to our business.
Lab49/ION adheres to an equal employment opportunity policy that prohibits discriminatory practices or harassment against applicants or employees based on any legally impermissible factor.