Highbrow LLC
Data Engineer
Highbrow LLC, Morris Plains, New Jersey, us, 07950
Job Title:
Data Engineer
Job ID: 2023-11994
Job Location: Morris Plains, NJ (remote to start)
Job Travel Location(s):
# Positions: 2
Employment Type: W2
Candidate Constraints:
Duration:Long Term
# of Layers:0
Work Eligibility:All Work Authorizations are Permitted
Key Technology:
Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop
Job Responsibilities:
Work with business and technical leadership to understand requirements.
Design to the requirements and document the designs.
Ability to write product-grade performant code for data extraction, transformations and loading using Spark, Py-Spark
Do data modeling as needed for the requirements.
Write performant queries using Teradata SQL, Hive SQL and Spark SQL against Teradata and Hive
Implementing dev-ops pipelines to deploy code artifacts on to the designated platform/servers like AWS or Hadoop Edge Nodes
Implement Hadoop job orchestration using Shell scripting, Apache Oozie, CA7 Enterprise Scheduler and Airflow
Troubleshooting the issues, providing effective solutions and jobs monitoring in the production environment
Participate in sprint planning sessions, refinement/story-grooming sessions, daily scrums, demos and retrospectives.
Skills and Experience Required:
Strong development experience in Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop
Experience of Ab Initio is a bonus.
Strong experience in writing complex and effective SQLs (using Teradata SQL, Hive SQL and Spark SQL) and Stored Procedures
Excellent work experience on Hadoop as data warehouse/Data Lake implementations
Experience in Agile and working knowledge on DevOps tools (Git, Jenkins, Artifactory)
Unix/Linux Shell scripting (KSH) and basic administration of Unix servers
CA7 Enterprise Scheduler
Experience with AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch)
Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure/AWS integration)
Experience in Jira and Confluence
Exercises considerable creativity, foresight, and judgment in conceiving, planning, and delivering initiatives.
Health care domain knowledge is a plus.
#J-18808-Ljbffr
Data Engineer
Job ID: 2023-11994
Job Location: Morris Plains, NJ (remote to start)
Job Travel Location(s):
# Positions: 2
Employment Type: W2
Candidate Constraints:
Duration:Long Term
# of Layers:0
Work Eligibility:All Work Authorizations are Permitted
Key Technology:
Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop
Job Responsibilities:
Work with business and technical leadership to understand requirements.
Design to the requirements and document the designs.
Ability to write product-grade performant code for data extraction, transformations and loading using Spark, Py-Spark
Do data modeling as needed for the requirements.
Write performant queries using Teradata SQL, Hive SQL and Spark SQL against Teradata and Hive
Implementing dev-ops pipelines to deploy code artifacts on to the designated platform/servers like AWS or Hadoop Edge Nodes
Implement Hadoop job orchestration using Shell scripting, Apache Oozie, CA7 Enterprise Scheduler and Airflow
Troubleshooting the issues, providing effective solutions and jobs monitoring in the production environment
Participate in sprint planning sessions, refinement/story-grooming sessions, daily scrums, demos and retrospectives.
Skills and Experience Required:
Strong development experience in Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop
Experience of Ab Initio is a bonus.
Strong experience in writing complex and effective SQLs (using Teradata SQL, Hive SQL and Spark SQL) and Stored Procedures
Excellent work experience on Hadoop as data warehouse/Data Lake implementations
Experience in Agile and working knowledge on DevOps tools (Git, Jenkins, Artifactory)
Unix/Linux Shell scripting (KSH) and basic administration of Unix servers
CA7 Enterprise Scheduler
Experience with AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch)
Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure/AWS integration)
Experience in Jira and Confluence
Exercises considerable creativity, foresight, and judgment in conceiving, planning, and delivering initiatives.
Health care domain knowledge is a plus.
#J-18808-Ljbffr