Data Engineer | Onsite | Dallas/Charlotte

Job Description

Job Description: Data Engineer

Role Overview: We are looking for a highly skilled

Data Engineer

to design, develop, and optimize data pipelines and infrastructure that enable data-driven decision-making.

Key Responsibilities:

Data Pipeline Development Design, build, and maintain

ETL/ELT pipelines

for data ingestion, transformation, and loading. Automate data workflows to ensure efficient data processing and availability. Integrate data from multiple sources such as APIs, streaming platforms, and databases. Data Infrastructure Management

Develop and optimize

data storage solutions , including

data lakes ,

data warehouses , and

databases . Collaborate with DevOps teams to manage cloud-based infrastructure for data pipelines. Ensure scalability, reliability, and performance of data systems to support business needs. Data Integration and Transformation

Implement real-time and batch data processing solutions using tools like

Apache Kafka ,

Spark , or

Flink . Transform raw data into structured formats to support analytics and reporting. Implement schema management, data partitioning, and version control. Data Quality and Governance

Monitor and improve

data quality , ensuring consistency and accuracy. Implement and enforce

data governance

practices, including metadata management, data lineage, and cataloging. Collaborate with stakeholders to establish standards for data validation and transformation. Cloud-Based Solutions

Leverage cloud platforms like

AWS ,

Azure , or

GCP

for building and deploying data infrastructure. Work with cloud-native tools (e.g.,

Azure Data Factory ,

AWS Glue ,

Google Cloud Dataflow ) for data integration and processing. Optimize cloud costs while ensuring high availability and performance. Collaboration and Documentation

Collaborate with data scientists, analysts, and business teams to understand data requirements. Document data workflows, pipeline designs, and infrastructure architecture. Provide technical support for data-related issues in production environments. Optimization and Performance

Identify and resolve bottlenecks in data pipelines and storage systems. Implement caching, indexing, and partitioning strategies to improve performance. Continuously monitor and optimize data workflows for scalability and efficiency. Key Qualifications:

Technical Expertise

Proficiency in programming languages such as

Python ,

Java , or

Scala

for data engineering tasks. Strong experience with

SQL

and database systems (e.g.,

PostgreSQL ,

MySQL ,

MongoDB ). Expertise in data processing tools like

Apache Spark ,

Hadoop , or

Flink . Familiarity with data orchestration tools like

Apache Airflow ,

Luigi , or

Prefect . Cloud and Big Data

Hands-on experience with cloud platforms ( AWS ,

Azure , or

GCP ) and their data services (e.g.,

Redshift ,

BigQuery ,

Azure Synapse ). Experience with streaming technologies like

Apache Kafka ,

RabbitMQ , or

Kinesis . Knowledge of data storage solutions such as

S3 ,

Azure Blob Storage , or

HDFS . Preferred Skills

Experience with data visualization tools like

Tableau ,

Power BI , or

Looker

is a plus. Knowledge of machine learning workflows and tools is desirable. Understanding of CI/CD pipelines for deploying data pipelines in production environments. Soft Skills

Strong problem-solving skills and attention to detail. Excellent communication and collaboration skills to work with cross-functional teams. Ability to prioritize tasks and manage time effectively in a fast-paced environment. Education and Experience:

Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field. 3+ years of experience in data engineering, big data, or a related field. Relevant certifications like

AWS Certified Data Analytics ,

Google Professional Data Engineer , or

Azure Data Engineer Associate

are preferred.

Compensation, Benefits and Duration

Minimum Compensation: USD 56,000 Maximum Compensation: USD 224,000 Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role. Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees. This position is not available for independent contractors No applications will be considered if received more than 120 days after the date of this post