Mudrasys
Data Engineer
Mudrasys, Pasco, Washington, us, 99302
Job title: Data Engineer
Job duties:
1. Data Pipeline Development: Design, develop, and maintain data pipelines to ingest, transform, and load data from various sources into data storage systems. This involves utilizing technologies such as Apache Kafka, Apache NiFi, or custom ETL (Extract, Transform, Load) processes to ensure the smooth and efficient flow of data.
2. Data Integration and Consolidation: Integrate and consolidate data from different sources, including databases, APIs, logs, and streaming platforms. Implement data extraction, transformation, and loading techniques to ensure data consistency and accuracy across systems. 3. Data Modelling and Architecture: Design and implement data models and database schemas that align with business requirements and ensure optimal data storage and retrieval performance. Apply data normalization and denormalization techniques as needed. 4. Data Warehouse and Storage Management: Manage and optimize data storage systems, such as data warehouses, data lakes, or cloud-based storage platforms. Ensure data integrity, security, and scalability, and implement appropriate backup and disaster recovery strategies. 5. Data Quality Assurance: Develop and implement data quality checks and validation processes to identify and resolve data quality issues. Implement data cleansing, standardization, and enrichment techniques to ensure data accuracy and consistency. 6. Data Transformation and Manipulation: Transform and manipulate data to meet specific business requirements, such as aggregations, filtering, or data formatting. Utilize tools and technologies like Apache Spark, SQL, or scripting languages to perform data transformations efficiently. 7. Performance Optimization: Identify and optimize performance bottlenecks in data processing and storage systems. Tune database queries, optimize data pipelines, and implement caching or indexing techniques to improve system efficiency and response times. 8. Data Security and Governance: Implement data security measures, access controls, and encryption mechanisms to protect sensitive data. Ensure compliance with data privacy regulations and company data governance policies. 9. Collaboration and Documentation: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective data solutions. Document data pipelines, systems, and processes for knowledge sharing and future reference. 10. Continuous Learning and Innovation: Stay updated with the latest trends, tools, and technologies in data engineering. Explore new technologies, evaluate their potential, and propose innovative solutions to improve data processing and storage capabilities.
Degree Requirement: This position requires, at a minimum, a bachelor's degree in computer science, computer information systems, information technology or a combination of education and experience equating to the U.S. equivalent of a bachelor's degree in one of the aforementioned subjects.
Work Schedule: Full Time, 40 Hours/Week.
Job duties:
1. Data Pipeline Development: Design, develop, and maintain data pipelines to ingest, transform, and load data from various sources into data storage systems. This involves utilizing technologies such as Apache Kafka, Apache NiFi, or custom ETL (Extract, Transform, Load) processes to ensure the smooth and efficient flow of data.
2. Data Integration and Consolidation: Integrate and consolidate data from different sources, including databases, APIs, logs, and streaming platforms. Implement data extraction, transformation, and loading techniques to ensure data consistency and accuracy across systems. 3. Data Modelling and Architecture: Design and implement data models and database schemas that align with business requirements and ensure optimal data storage and retrieval performance. Apply data normalization and denormalization techniques as needed. 4. Data Warehouse and Storage Management: Manage and optimize data storage systems, such as data warehouses, data lakes, or cloud-based storage platforms. Ensure data integrity, security, and scalability, and implement appropriate backup and disaster recovery strategies. 5. Data Quality Assurance: Develop and implement data quality checks and validation processes to identify and resolve data quality issues. Implement data cleansing, standardization, and enrichment techniques to ensure data accuracy and consistency. 6. Data Transformation and Manipulation: Transform and manipulate data to meet specific business requirements, such as aggregations, filtering, or data formatting. Utilize tools and technologies like Apache Spark, SQL, or scripting languages to perform data transformations efficiently. 7. Performance Optimization: Identify and optimize performance bottlenecks in data processing and storage systems. Tune database queries, optimize data pipelines, and implement caching or indexing techniques to improve system efficiency and response times. 8. Data Security and Governance: Implement data security measures, access controls, and encryption mechanisms to protect sensitive data. Ensure compliance with data privacy regulations and company data governance policies. 9. Collaboration and Documentation: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective data solutions. Document data pipelines, systems, and processes for knowledge sharing and future reference. 10. Continuous Learning and Innovation: Stay updated with the latest trends, tools, and technologies in data engineering. Explore new technologies, evaluate their potential, and propose innovative solutions to improve data processing and storage capabilities.
Degree Requirement: This position requires, at a minimum, a bachelor's degree in computer science, computer information systems, information technology or a combination of education and experience equating to the U.S. equivalent of a bachelor's degree in one of the aforementioned subjects.
Work Schedule: Full Time, 40 Hours/Week.