Atlantic Partners
Azure Data Engineer
Atlantic Partners, Columbia, South Carolina, us, 29228
Overview:A Data Engineer is responsible for designing, building, and maintaining data architectures and pipelines that collect, process, and store large datasets for various analytical and operational purposes. They play a critical role in ensuring that data is available, clean, secure, and ready for analysis and reporting. Data engineers work closely with data scientists, analysts, and other stakeholders to optimize data flow and ensure that the infrastructure supports the organization's data needs.Key Responsibilities:Data Pipeline Development:
Design, build, and maintain scalable and efficient data pipelines that automate the extraction, transformation, and loading (ETL) of data from multiple sources into data warehouses, data lakes, or other storage solutions.Develop both batch and real-time data processing solutions to handle structured, semi-structured, and unstructured data.Ensure that data pipelines are reliable, resilient, and efficient in terms of performance and cost.
Data Infrastructure & Storage Management:
Architect and implement scalable and fault-tolerant data storage systems, such as relational databases (e.g., MySQL, PostgreSQL), NoSQL databases (e.g., MongoDB, Cassandra), and cloud-based data warehouses (e.g., AWS Redshift, Google BigQuery, Snowflake).Optimize data storage for both structured and unstructured data, ensuring efficient querying and access for business intelligence (BI) and analytics.
Data Integration:
Integrate data from multiple sources, such as internal databases, APIs, third-party applications, logs, and external data sources, into unified platforms for analytics and reporting.Implement tools and solutions for data ingestion from disparate systems, ensuring the consistency and accuracy of data across the organization.
Data Quality, Governance, and Security:
Ensure the accuracy, completeness, and consistency of data by implementing robust data validation, cleansing, and auditing processes.Work closely with data governance teams to define data standards, metadata management, and access policies.Ensure data security and compliance with relevant regulations (e.g., GDPR, CCPA) by implementing encryption, access controls, and secure data-sharing practices.
Collaboration with Data Teams:
Collaborate with data scientists, analysts, and business stakeholders to understand their data needs and ensure the data infrastructure supports their goals.Work with cross-functional teams to improve the overall data architecture and suggest enhancements that drive better business outcomes.
Performance Tuning & Optimization:
Continuously monitor and tune data systems for performance, ensuring fast and reliable data access.Optimize query performance and storage costs, especially in cloud environments, by leveraging appropriate indexing, partitioning, and caching techniques.
Monitoring and Maintenance:
Develop and implement monitoring tools to track the performance and reliability of data pipelines, data storage, and other infrastructure components.Troubleshoot data-related issues, including pipeline failures, bottlenecks, and data anomalies, to ensure business continuity.Provide ongoing maintenance and support for data systems, ensuring uptime and reliability.
Adoption of New Technologies:
Stay up to date with emerging technologies and trends in data engineering, such as cloud-native architectures, distributed data processing, and big data frameworks (e.g., Hadoop, Spark).Proactively recommend and implement new tools and technologies that can enhance the organization's data engineering capabilities.
#J-18808-Ljbffr
Design, build, and maintain scalable and efficient data pipelines that automate the extraction, transformation, and loading (ETL) of data from multiple sources into data warehouses, data lakes, or other storage solutions.Develop both batch and real-time data processing solutions to handle structured, semi-structured, and unstructured data.Ensure that data pipelines are reliable, resilient, and efficient in terms of performance and cost.
Data Infrastructure & Storage Management:
Architect and implement scalable and fault-tolerant data storage systems, such as relational databases (e.g., MySQL, PostgreSQL), NoSQL databases (e.g., MongoDB, Cassandra), and cloud-based data warehouses (e.g., AWS Redshift, Google BigQuery, Snowflake).Optimize data storage for both structured and unstructured data, ensuring efficient querying and access for business intelligence (BI) and analytics.
Data Integration:
Integrate data from multiple sources, such as internal databases, APIs, third-party applications, logs, and external data sources, into unified platforms for analytics and reporting.Implement tools and solutions for data ingestion from disparate systems, ensuring the consistency and accuracy of data across the organization.
Data Quality, Governance, and Security:
Ensure the accuracy, completeness, and consistency of data by implementing robust data validation, cleansing, and auditing processes.Work closely with data governance teams to define data standards, metadata management, and access policies.Ensure data security and compliance with relevant regulations (e.g., GDPR, CCPA) by implementing encryption, access controls, and secure data-sharing practices.
Collaboration with Data Teams:
Collaborate with data scientists, analysts, and business stakeholders to understand their data needs and ensure the data infrastructure supports their goals.Work with cross-functional teams to improve the overall data architecture and suggest enhancements that drive better business outcomes.
Performance Tuning & Optimization:
Continuously monitor and tune data systems for performance, ensuring fast and reliable data access.Optimize query performance and storage costs, especially in cloud environments, by leveraging appropriate indexing, partitioning, and caching techniques.
Monitoring and Maintenance:
Develop and implement monitoring tools to track the performance and reliability of data pipelines, data storage, and other infrastructure components.Troubleshoot data-related issues, including pipeline failures, bottlenecks, and data anomalies, to ensure business continuity.Provide ongoing maintenance and support for data systems, ensuring uptime and reliability.
Adoption of New Technologies:
Stay up to date with emerging technologies and trends in data engineering, such as cloud-native architectures, distributed data processing, and big data frameworks (e.g., Hadoop, Spark).Proactively recommend and implement new tools and technologies that can enhance the organization's data engineering capabilities.
#J-18808-Ljbffr