Diverse Lynx
Data Engineer
Diverse Lynx, O Fallon, Missouri, United States, 63366
Job Description -• Minimum 5+ years of IT experience and minimum 3+ years of hands-on experience in Databricks and pyspark with validated knowledge in designing database models to store structured & unstructured data efficiently and in creating effective data tools for analytics experts along with following:• Familiarity with Lakehouse Architecture with Delta Tables - Delta tables, Schema Evolution, ACID compatibility, History capture etc.• Integrating Databricks with various ETL/Orchestration tools and perform required operations.• Python is Must• CI/CD Pipeline with AWS• SQL and SPARK with Date Engineering background• Expertise in Spark ecosystems, Data frame API, Data set API, RDD APIs & Spark SQL.• Expertise in workspace and cluster configuration, performance tuning and log monitoring.• Experience with structured streaming along with streaming frameworks.• Hands on experience with Python, Scala and SQL.• Create and maintain optimal data pipeline architecture with knowledge in best practices from Databricks around Cluster setup/Development/Deployment• Strong understanding of Data Modelling and defining conceptual logical and physical data models.• Experience in building Data Engineering solutions using AWS native stack• Experience in managing various client relationships to get consensus on solutions.• Nice to have AI/Client skill and any architect level certification from Databricks
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.