Beacon Hill
AI Big Data Engineer
Beacon Hill, Troy, Michigan, United States, 48083
To Apply for this Job Click HereSummary:As an AI Big Data Engineer, you will focus on creating a Unified Data Platform. You will design, develop, and maintain data pipelines, data lakes, and data platforms to support analytics and business intelligence needs. This role involves working with cutting-edge technologies and tools like Spark, Kafka, AWS, Azure, and Kubernetes to manage large-scale and complex data challenges. Collaboration with full stack developers, data scientists, analysts, and stakeholders are essential to ensure data quality, reliability, and usability. Comfort with handling huge datasets is a must.Main Responsibilities:
Build automated pipelines to extract and process data from legacy platforms (predominantly SQL Server), using tools like stored procedures, Glue processing, etc.Implement data-related business logic on modern data platforms, such as AWS Glue, Databricks, and Azure, using best practices and industry standards.Create vector databases, data marts, and data models to support them.Optimize and monitor performance, reliability, and security of data systems and processes.Integrate and transform data from various sources and formats, such as structured, unstructured, streaming, and batch.Develop and maintain data quality checks, tests, and documentation.Support data analysis, reporting, and visualization using tools like SQL, Python, Tableau, and Quicksight.Research and evaluate new data technologies and trends to enhance data solutions and capabilities.Qualifications and Skills:
Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or a related field.At least 5 years of experience in data engineering or a similar role (DBA experience is a plus).Experience with big data frameworks and tools such as Spark, Hadoop, Kafka, and Hive.Expert in SQL, with knowledge of efficient query and schema design, DDL, data modeling, and stored procedures.Proficient in at least one programming language such as Python, Go, or Java.Experience with CI/CD, containerization (e.g., Docker, Kubernetes), and orchestration (e.g., Airflow).Experience building production systems with modern ETL/ELT and data platforms such as AWS Glue, Databricks, Snowflake, and Azure Cognitive Search.Experience deploying data infrastructure on cloud platforms (AWS, Azure, or GCP).Strong knowledge of data quality, governance, and security principles and practices.Excellent communication, collaboration, and problem-solving skills.
1394392_1728337788To Apply for this Job Click Here
Build automated pipelines to extract and process data from legacy platforms (predominantly SQL Server), using tools like stored procedures, Glue processing, etc.Implement data-related business logic on modern data platforms, such as AWS Glue, Databricks, and Azure, using best practices and industry standards.Create vector databases, data marts, and data models to support them.Optimize and monitor performance, reliability, and security of data systems and processes.Integrate and transform data from various sources and formats, such as structured, unstructured, streaming, and batch.Develop and maintain data quality checks, tests, and documentation.Support data analysis, reporting, and visualization using tools like SQL, Python, Tableau, and Quicksight.Research and evaluate new data technologies and trends to enhance data solutions and capabilities.Qualifications and Skills:
Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or a related field.At least 5 years of experience in data engineering or a similar role (DBA experience is a plus).Experience with big data frameworks and tools such as Spark, Hadoop, Kafka, and Hive.Expert in SQL, with knowledge of efficient query and schema design, DDL, data modeling, and stored procedures.Proficient in at least one programming language such as Python, Go, or Java.Experience with CI/CD, containerization (e.g., Docker, Kubernetes), and orchestration (e.g., Airflow).Experience building production systems with modern ETL/ELT and data platforms such as AWS Glue, Databricks, Snowflake, and Azure Cognitive Search.Experience deploying data infrastructure on cloud platforms (AWS, Azure, or GCP).Strong knowledge of data quality, governance, and security principles and practices.Excellent communication, collaboration, and problem-solving skills.
1394392_1728337788To Apply for this Job Click Here