Akraya
Sr. Data Engineer :
Akraya, New York, New York, us, 10261
Primary Skills: SQL (Expert), Scala (Advanced), Python (Advanced), Spark (Expert), Airflow (Expert), AWS (Intermediate)Contract Type: W2Duration: 6 MonthsLocation: New York, NY, Burbank CA, or Seattle WA, Santa Monica CA (Hybrid)Pay Range: $70.00 to 80.00 Per hourJOB SUMMARY:A leading fortune 500 Media and Entertainment company is looking for a Sr. Data Engineer with experience in Scala and Python that abstract complex business logic to allow consistent functionality across all data pipelines across the Data organization who can Collaborate with product managers, architects, and other engineers to drive the success of the Core Data platform and contribute to developing and documenting both internal and external standards and best practices for pipeline configurations, naming conventions, partitioning strategies, and more. Ensure high operational efficiency and quality of the Core Data platform datasets to ensure our solutions meet SLAs and project reliability and accuracy to all our stakeholders (Engineering, Data Science, Operations, and Analytics teams.JOB RESPONSIBILITIES:
Contribute to maintaining, updating, and expanding existing Core Data platform data pipelines in Scala and Python / Spark while maintaining strict uptime SLAs.Extend functionality of current Core Data platform offerings, including metadata parsing, extending the Metastore API, and building new integrations with APIs both internal and external to the Data organizationImplement Ingestion of new batch and streaming data pipelines using Scala, Databricks, and AirflowImplementation shared libraries in Scala and Python that abstract complex business logic to allow consistent functionality across all data pipelines across the Data organizationTech stack includes Airflow, Spark, Databricks, Delta Lake, Snowflake, Scala, PythonImplement the Lakehouse architecture, working with customers, partners, and stakeholders to shift towards a Lakehouse centric data platform.
JOB REQUIREMENTS:
5+ years of data engineering experience developing large data pipelinesDeep Understanding of AWS or other cloud providers as well as infrastructure as codeExperience in Data Modeling techniques and Data Warehousing standard methodologies and practicesStrong fundamental Scala and Python programming skills.Strong SQL skills and ability to create queries to analyze complex datasetsHands-on production environment experience with distributed processing systems such as Spark and Airflow for creating and maintaining data pipelinesSome scripting language experience-Preferred.
ABOUT AKRAYAAkraya is an award-winning IT staffing firm consistently recognized for our commitment to excellence and a thriving work environment. Most recently, we were recognized Inc's Best Workplaces 2024 and Silicon Valley's Best Places to Work by the San Francisco Business Journal (2024) and Glassdoor's Best Places to Work (2023 & 2022)!Industry Leaders in IT StaffingAs staffing solutions providers for Fortune 100 companies, Akraya's industry recognitions solidify our leadership position in the IT staffing space. We don't just connect you with great jobs, we connect you with a workplace that inspires!Join Akraya Today!Let us lead you to your dream career and experience the Akraya difference. Browse our open positions and join our team!
Contribute to maintaining, updating, and expanding existing Core Data platform data pipelines in Scala and Python / Spark while maintaining strict uptime SLAs.Extend functionality of current Core Data platform offerings, including metadata parsing, extending the Metastore API, and building new integrations with APIs both internal and external to the Data organizationImplement Ingestion of new batch and streaming data pipelines using Scala, Databricks, and AirflowImplementation shared libraries in Scala and Python that abstract complex business logic to allow consistent functionality across all data pipelines across the Data organizationTech stack includes Airflow, Spark, Databricks, Delta Lake, Snowflake, Scala, PythonImplement the Lakehouse architecture, working with customers, partners, and stakeholders to shift towards a Lakehouse centric data platform.
JOB REQUIREMENTS:
5+ years of data engineering experience developing large data pipelinesDeep Understanding of AWS or other cloud providers as well as infrastructure as codeExperience in Data Modeling techniques and Data Warehousing standard methodologies and practicesStrong fundamental Scala and Python programming skills.Strong SQL skills and ability to create queries to analyze complex datasetsHands-on production environment experience with distributed processing systems such as Spark and Airflow for creating and maintaining data pipelinesSome scripting language experience-Preferred.
ABOUT AKRAYAAkraya is an award-winning IT staffing firm consistently recognized for our commitment to excellence and a thriving work environment. Most recently, we were recognized Inc's Best Workplaces 2024 and Silicon Valley's Best Places to Work by the San Francisco Business Journal (2024) and Glassdoor's Best Places to Work (2023 & 2022)!Industry Leaders in IT StaffingAs staffing solutions providers for Fortune 100 companies, Akraya's industry recognitions solidify our leadership position in the IT staffing space. We don't just connect you with great jobs, we connect you with a workplace that inspires!Join Akraya Today!Let us lead you to your dream career and experience the Akraya difference. Browse our open positions and join our team!