Big Data Engineer

The client Corporation is currently seeking an experienced Big Data Engineer. The successful candidate must have Big Data engineering experience and must demonstrate an affinity for working with others to create successful solutions. Join a smart, highly skilled team with a passion for technology, where you will work on our state-of-the-art Big Data Platforms. They must be a very good communicator, both written and verbal, and have some experience working with business areas to translate their business data needs and data questions into project requirements. The candidate will participate in all phases of the Data Engineering life cycle and will independently and collaboratively write project requirements, architect solutions, and perform data ingestion development and support duties.Skills and Experience:Required:

6+ years of overall IT experience3+ years of experience with high-velocity high-volume stream processing: Apache Kafka and Spark StreamingExperience with real-time data processing and streaming techniques using Spark structured streaming and KafkaDeep knowledge of troubleshooting and tuning Spark applications3+ years of experience with data ingestion from Message Queues (Tibco, IBM, etc.) and different file formats across different platforms like JSON, XML, CSV3+ years of experience with Big Data tools/technologies like Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, HDFS3+ years of experience building, testing, and optimizing ‘Big Data’ data ingestion pipelines, architectures, and data sets2+ years of experience with Python (and/or Scala) and PySpark/Scala-Spark3+ years of experience with Cloud platforms e.g. AWS, Google Cloud Platform, etc.3+ years of experience with database solutions like Kudu/Impala, Delta Lake, Snowflake, or BigQuery2+ years of experience with NoSQL databases, including HBASE and/or CassandraExperience in successfully building and deploying a new data platform on Azure/AWSExperience in Azure/AWS Serverless technologies, like S3, Kinesis/MSK, Lambda, and GlueStrong knowledge of Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ SeriesExperience with Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity CatalogKnowledge of Unix/Linux platform and shell scripting is a mustStrong analytical and problem-solving skillsPreferred:

Strong SQL skills with the ability to write intermediate complexity queriesStrong understanding of Relational & Dimensional modelingExperience with GIT code versioning softwareExperience with REST API and Web ServicesGood business analyst and requirements gathering/writing skills

#J-18808-Ljbffr