eTeam
Sr Big data Engineer - Hybrid
eTeam, Plano, Texas, us, 75086
Job Details:
Must Have Skills: Spark Scala Databricks
Detailed Job Description: 12+ years industry experience building and supporting large-scale cloud systems (AWS/Azure). • Experience with Databricks, Spark and working knowledge of DBT for data processing and transformation. • A working understanding of Data Warehousing concepts of Data Vault, Dimensional modelling, OLAP design. • Direct experience of building data pipelines using technology such as Azure Data Factory and Apache Spark (preferably in Databricks). • Extensive experience with CI/CD platforms such as GitLab CI, GitHub Actions, AWS CodeBuild, Azure Pipelines, and Jenkins. • Skilled in scripting languages including Python, Bash, and PowerShell. • Familiarity with log and monitoring solutions • Experience in utilising development containers, unit testing/code quality review/linting - standard best practise. • Proficiency in Python and SQL for data transformation and querying. • Strong expertise in version control systems. • Familiarity with containerization and orchestration technologies • Strong problem-solving, troubleshooting, communication and collaboration skills.
Must Have Skills: Spark Scala Databricks
Detailed Job Description: 12+ years industry experience building and supporting large-scale cloud systems (AWS/Azure). • Experience with Databricks, Spark and working knowledge of DBT for data processing and transformation. • A working understanding of Data Warehousing concepts of Data Vault, Dimensional modelling, OLAP design. • Direct experience of building data pipelines using technology such as Azure Data Factory and Apache Spark (preferably in Databricks). • Extensive experience with CI/CD platforms such as GitLab CI, GitHub Actions, AWS CodeBuild, Azure Pipelines, and Jenkins. • Skilled in scripting languages including Python, Bash, and PowerShell. • Familiarity with log and monitoring solutions • Experience in utilising development containers, unit testing/code quality review/linting - standard best practise. • Proficiency in Python and SQL for data transformation and querying. • Strong expertise in version control systems. • Familiarity with containerization and orchestration technologies • Strong problem-solving, troubleshooting, communication and collaboration skills.