Simpleelabs
Senior Big Data Engineer
Simpleelabs, Sunnyvale, California, United States, 94087
Role: Senior Big Data Engineer
Location: Sunnyvale, CA / Dallas, TX - Hybrid Role (Locals Only)
Job Type:
Contract
Experience:
12+ Years
Tech stack: Google cloud, HDFS, SPARK, Scala, Python (optional), Automic/Airflow, BigQuery, Kafka, API
Big Data Engineer:
Must Have:
Proficiency in managing and manipulating huge datasets in the order of terabytes (TB) is essential
Expertise in big data technologies like Hadoop, Apache Spark (Scala preferred), Apache Hive, or similar frameworks on the cloud (GCP preferred, AWS, Azure etc.) to build batch data pipelines with strong focus on optimization, SLA adherence and fault tolerance.
Expertise in building idempotent workflows using orchestrators like Automic, Airflow, Luigi etc.
Expertise in writing SQL to analyze, optimize, profile data preferably in BigQuery or SPARK SQL
Strong data modeling skills are necessary for designing a schema that can accommodate the evolution of data sources and facilitate seamless data joins across various datasets
Ability to work directly with stakeholders to understand data requirements and translate that to pipeline development / data solution work
Strong analytical and problem-solving skills are crucial for identifying and resolving issues that may arise during the data integration and schema evolution process
Ability to move at rapid pace with quality and start delivering with minimal ramp up time will be crucial to succeed in this initiative
Effective communication and collaboration skills are necessary for working in a team environment and coordinating efforts between different stakeholders involved in the project
Nice to have:
Experience building complex near real time (NRT) streaming data pipelines using Apache Kafka, Spark streaming, Kafka Connect with a strong focus on stability, scalability and SLA adherence.
Good understanding of REST APIs – working knowledge on Apache Druid, Redis, Elastic search, GraphQL or similar technologies.Understanding of API contracts, building telemetry, stress testing etc.
Exposure in developing reports/dashboards using Looker/Tableau
Experience in the e-Commerce domain.
#J-18808-Ljbffr
Location: Sunnyvale, CA / Dallas, TX - Hybrid Role (Locals Only)
Job Type:
Contract
Experience:
12+ Years
Tech stack: Google cloud, HDFS, SPARK, Scala, Python (optional), Automic/Airflow, BigQuery, Kafka, API
Big Data Engineer:
Must Have:
Proficiency in managing and manipulating huge datasets in the order of terabytes (TB) is essential
Expertise in big data technologies like Hadoop, Apache Spark (Scala preferred), Apache Hive, or similar frameworks on the cloud (GCP preferred, AWS, Azure etc.) to build batch data pipelines with strong focus on optimization, SLA adherence and fault tolerance.
Expertise in building idempotent workflows using orchestrators like Automic, Airflow, Luigi etc.
Expertise in writing SQL to analyze, optimize, profile data preferably in BigQuery or SPARK SQL
Strong data modeling skills are necessary for designing a schema that can accommodate the evolution of data sources and facilitate seamless data joins across various datasets
Ability to work directly with stakeholders to understand data requirements and translate that to pipeline development / data solution work
Strong analytical and problem-solving skills are crucial for identifying and resolving issues that may arise during the data integration and schema evolution process
Ability to move at rapid pace with quality and start delivering with minimal ramp up time will be crucial to succeed in this initiative
Effective communication and collaboration skills are necessary for working in a team environment and coordinating efforts between different stakeholders involved in the project
Nice to have:
Experience building complex near real time (NRT) streaming data pipelines using Apache Kafka, Spark streaming, Kafka Connect with a strong focus on stability, scalability and SLA adherence.
Good understanding of REST APIs – working knowledge on Apache Druid, Redis, Elastic search, GraphQL or similar technologies.Understanding of API contracts, building telemetry, stress testing etc.
Exposure in developing reports/dashboards using Looker/Tableau
Experience in the e-Commerce domain.
#J-18808-Ljbffr