Reuben Cooley, Inc.
Job Description
We are seeking a highly skilled and experienced Data Engineer with 6+ years of experience to join our team. The ideal candidate has a strong background in Python (Pandas, NumPy, PySpark, PyOdbc, Py MsSQL, Requests, Boto3, Simple Salesforce, Json), Databricks, Snowflake, ETL development, data modeling, data lineage, and data governance. You will be responsible for designing, building, and optimizing large-scale data pipelines and ensuring data quality, security, and compliance.
Key Responsibilities:
Develop, maintain, and optimize ETL pipelines using Databricks, Snowflake, Apache Spark, and Delta Lake. Implement data partitioning, optimization, and migration strategies. Work on data modeling, data lineage, and governance to ensure data integrity and compliance. Design and manage RBAC, Unity Catalog, Encryption, and Compliance in Databricks and Snowflake. Develop and execute workflows, notebooks, and automation using Databricks, Apache Airflow, AWS Glue, Azure Data Factory, Talend, or Alteryx. Manage database design and queries in MSSQL, MySQL, DynamoDB, and MongoDB. Work with cloud platforms such as AWS or Azure, integrating services like Boto3 for AWS, Data Factory for Azure, and Delta Share for data exchange. Collaborate with data scientists, analysts, and business stakeholders to design efficient data solutions. Implement and maintain data encryption, access control, and governance frameworks. Utilize Collibra and Hackolade for data cataloging, governance, and metadata management.
Required Skills & Experience:
6+ years of experience in Data Engineering, Data Architecture, or related fields. Strong Python skills, including experience with Pandas, NumPy, PySpark, PyOdbc, PyMsSQL, Requests, and Boto3. Hands-on experience with Databricks, Snowflake, and Apache Spark. Expertise in SQL (MSSQL, MySQL) and NoSQL databases (DynamoDB, MongoDB). Experience working with ETL tools such as Apache Airflow, AWS Glue, Azure Data Factory, Talend, or Alteryx. Strong knowledge of data modeling, lineage, governance, and compliance. Experience with RBAC, Unity Catalog, Encryption & Compliance frameworks. Proficiency in data migration strategies and pipeline optimization. Familiarity with Collibra, Hackolade, and metadata management tools. Strong problem-solving skills, attention to detail, and ability to work in a fast-paced environment.
Preferred Qualifications:
Experience working with large-scale enterprise environments. Certification in Databricks, Snowflake, AWS, or Azure. Knowledge of Machine Learning, AI pipelines, or advanced analytics.
Key Responsibilities:
Develop, maintain, and optimize ETL pipelines using Databricks, Snowflake, Apache Spark, and Delta Lake. Implement data partitioning, optimization, and migration strategies. Work on data modeling, data lineage, and governance to ensure data integrity and compliance. Design and manage RBAC, Unity Catalog, Encryption, and Compliance in Databricks and Snowflake. Develop and execute workflows, notebooks, and automation using Databricks, Apache Airflow, AWS Glue, Azure Data Factory, Talend, or Alteryx. Manage database design and queries in MSSQL, MySQL, DynamoDB, and MongoDB. Work with cloud platforms such as AWS or Azure, integrating services like Boto3 for AWS, Data Factory for Azure, and Delta Share for data exchange. Collaborate with data scientists, analysts, and business stakeholders to design efficient data solutions. Implement and maintain data encryption, access control, and governance frameworks. Utilize Collibra and Hackolade for data cataloging, governance, and metadata management.
Required Skills & Experience:
6+ years of experience in Data Engineering, Data Architecture, or related fields. Strong Python skills, including experience with Pandas, NumPy, PySpark, PyOdbc, PyMsSQL, Requests, and Boto3. Hands-on experience with Databricks, Snowflake, and Apache Spark. Expertise in SQL (MSSQL, MySQL) and NoSQL databases (DynamoDB, MongoDB). Experience working with ETL tools such as Apache Airflow, AWS Glue, Azure Data Factory, Talend, or Alteryx. Strong knowledge of data modeling, lineage, governance, and compliance. Experience with RBAC, Unity Catalog, Encryption & Compliance frameworks. Proficiency in data migration strategies and pipeline optimization. Familiarity with Collibra, Hackolade, and metadata management tools. Strong problem-solving skills, attention to detail, and ability to work in a fast-paced environment.
Preferred Qualifications:
Experience working with large-scale enterprise environments. Certification in Databricks, Snowflake, AWS, or Azure. Knowledge of Machine Learning, AI pipelines, or advanced analytics.