JobRialto
Data Engineer/Architect
JobRialto, Plainsboro, New Jersey, us, 08536
Job Summary:
The Data Engineer / Architect will leverage 14 years of experience in data engineering, analytics, and cloud platforms (AWS, GCP, Azure) to design and develop data pipelines and analytical solutions. The role requires extensive expertise in working with modern big data technologies, ETL tools, cloud data services, and advanced databases. This position will involve building end-to-end solutions to handle large datasets, ensuring efficient data integration, migration, and transformation processes.
Key Responsibilities:
• Design and develop scalable data pipelines and analytical solutions using Python, Spark, Hadoop, Java, Scala, and SQL.
• Leverage Apache Hadoop ecosystem tools such as HDFS, Hive, Spark Streaming, Kafka, and MapReduce for data engineering.
• Develop, integrate, and optimize ETL solutions using Informatica Cloud (IICS), Power Automate, and Power Apps.
• Utilize Azure Data Factory, Azure Databricks, and Synapse Analytics for building cloud-native data pipelines.
• Migrate and transform data from various sources into data warehouses using Informatica PowerCenter and PL/SQL conversion.
• Work with NoSQL databases, including MongoDB, HBase, and Druid, to manage structured and unstructured data.
• Implement data ingestion pipelines using Sqoop, Kafka, and Flume.
• Develop dashboards and visualizations using Tableau for enhanced data insights.
• Collaborate with stakeholders to define data strategies and optimize data solutions.
• Conduct performance tuning and ensure high availability of data infrastructure.
Required Qualifications:
• 14+ years of experience in data engineering and analytics.
• Expertise with cloud platforms such as AWS, GCP, and Azure.
• Proficient in tools like PySpark, Hive, Druid, and Hadoop components (HDFS, Spark, MapReduce).
• Strong experience in Java, Scala, and SQL for data processing.
• Extensive ETL development experience using Informatica (IICS, PowerCenter).
• Proficient in data warehousing, data migration, and integration processes.
• Experience working with Azure services like Data Factory, Databricks, and Synapse Analytics.
• Expertise in using Kafka, Sqoop, Flume, and Pig for data ingestion.
• Hands-on experience with NoSQL databases like MongoDB and HBase.
• Proficient in data visualization using Tableau.
Preferred Qualifications:
• Experience in building data solutions with Power Automate and Power Apps.
• Familiarity with Oozie and Yarn for job scheduling and cluster resource management.
• Experience in developing and optimizing Spark Streaming pipelines.
• Knowledge of modern data lake architecture and big data frameworks.
Certifications:
• AWS Certified Data Analytics Specialty or equivalent.
• Microsoft Certified: Azure Data Engineer Associate.
• Informatica Certified Professional (ICP) in Data Integration.
Education:
Bachelors Degree
The Data Engineer / Architect will leverage 14 years of experience in data engineering, analytics, and cloud platforms (AWS, GCP, Azure) to design and develop data pipelines and analytical solutions. The role requires extensive expertise in working with modern big data technologies, ETL tools, cloud data services, and advanced databases. This position will involve building end-to-end solutions to handle large datasets, ensuring efficient data integration, migration, and transformation processes.
Key Responsibilities:
• Design and develop scalable data pipelines and analytical solutions using Python, Spark, Hadoop, Java, Scala, and SQL.
• Leverage Apache Hadoop ecosystem tools such as HDFS, Hive, Spark Streaming, Kafka, and MapReduce for data engineering.
• Develop, integrate, and optimize ETL solutions using Informatica Cloud (IICS), Power Automate, and Power Apps.
• Utilize Azure Data Factory, Azure Databricks, and Synapse Analytics for building cloud-native data pipelines.
• Migrate and transform data from various sources into data warehouses using Informatica PowerCenter and PL/SQL conversion.
• Work with NoSQL databases, including MongoDB, HBase, and Druid, to manage structured and unstructured data.
• Implement data ingestion pipelines using Sqoop, Kafka, and Flume.
• Develop dashboards and visualizations using Tableau for enhanced data insights.
• Collaborate with stakeholders to define data strategies and optimize data solutions.
• Conduct performance tuning and ensure high availability of data infrastructure.
Required Qualifications:
• 14+ years of experience in data engineering and analytics.
• Expertise with cloud platforms such as AWS, GCP, and Azure.
• Proficient in tools like PySpark, Hive, Druid, and Hadoop components (HDFS, Spark, MapReduce).
• Strong experience in Java, Scala, and SQL for data processing.
• Extensive ETL development experience using Informatica (IICS, PowerCenter).
• Proficient in data warehousing, data migration, and integration processes.
• Experience working with Azure services like Data Factory, Databricks, and Synapse Analytics.
• Expertise in using Kafka, Sqoop, Flume, and Pig for data ingestion.
• Hands-on experience with NoSQL databases like MongoDB and HBase.
• Proficient in data visualization using Tableau.
Preferred Qualifications:
• Experience in building data solutions with Power Automate and Power Apps.
• Familiarity with Oozie and Yarn for job scheduling and cluster resource management.
• Experience in developing and optimizing Spark Streaming pipelines.
• Knowledge of modern data lake architecture and big data frameworks.
Certifications:
• AWS Certified Data Analytics Specialty or equivalent.
• Microsoft Certified: Azure Data Engineer Associate.
• Informatica Certified Professional (ICP) in Data Integration.
Education:
Bachelors Degree