Insight Global
GCP Data Engineer
Insight Global, Dearborn, Michigan, United States, 48120
Job Description
Day to Day:
The Data Engineer will join our automotive clients team working in one of the worlds largest data lakes. This teams current goal is to move their data from on-prem to GCP by utilizing Airflow and BigQuery. You will work with on-prem data sources (Hadoop, SQL Server, Teradata, etc.), understand the data model & business rules behind the data, and build data pipelines (with GCP) using Airflow for workflows and pipeline management. You will also partner with IT source teams to ingest data from various sources into a big data environment and streamline data ingestion processes. This data will be landed in GCP BigQuery.
Responsibilities:
Design data models for data warehouse with standards for consistency and integrity
Architect and develop data warehousing processes
Develop and maintain data dictionaries/metadata management processes for source data schema
Streamline data ingestion processes
Develop scalable, re-usable and metadata-driven solutions for data landing and transformation
Develop repeatable processes that accelerate delivery of user request fulfilment
Monitor data pipelines from sources to a big data environment
Compensation:
$65HR to $70HR
Exact compensation may vary based on several factors, including skills, experience, and education.
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com .
To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ .
Skills and Requirements
Qualifications:
8+ years of experience as Data Engineer
Strong experience working in Google Cloud Platform (GCP)
o Development in Cloud Run/Cloud Function, Pub/Sub Integration
o BigQuery development/operations
o IAM, Cloud Logging, Monitoring principles, operations
Framworks- no open soruce tools. Injguestion
Expert level Python programming experience
Strong understand on CI/CD- Teraform- Infrastructure Tekton-Deployment
Plus:
· Knowledge of Dataflow is a plus
· Knowledge in Terraform development is a plus
Knowledge of Java null
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to HR@insightglobal.com.
Day to Day:
The Data Engineer will join our automotive clients team working in one of the worlds largest data lakes. This teams current goal is to move their data from on-prem to GCP by utilizing Airflow and BigQuery. You will work with on-prem data sources (Hadoop, SQL Server, Teradata, etc.), understand the data model & business rules behind the data, and build data pipelines (with GCP) using Airflow for workflows and pipeline management. You will also partner with IT source teams to ingest data from various sources into a big data environment and streamline data ingestion processes. This data will be landed in GCP BigQuery.
Responsibilities:
Design data models for data warehouse with standards for consistency and integrity
Architect and develop data warehousing processes
Develop and maintain data dictionaries/metadata management processes for source data schema
Streamline data ingestion processes
Develop scalable, re-usable and metadata-driven solutions for data landing and transformation
Develop repeatable processes that accelerate delivery of user request fulfilment
Monitor data pipelines from sources to a big data environment
Compensation:
$65HR to $70HR
Exact compensation may vary based on several factors, including skills, experience, and education.
Benefit packages for this role will start on the 31st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com .
To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/ .
Skills and Requirements
Qualifications:
8+ years of experience as Data Engineer
Strong experience working in Google Cloud Platform (GCP)
o Development in Cloud Run/Cloud Function, Pub/Sub Integration
o BigQuery development/operations
o IAM, Cloud Logging, Monitoring principles, operations
Framworks- no open soruce tools. Injguestion
Expert level Python programming experience
Strong understand on CI/CD- Teraform- Infrastructure Tekton-Deployment
Plus:
· Knowledge of Dataflow is a plus
· Knowledge in Terraform development is a plus
Knowledge of Java null
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion,sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military oruniformed service member status, or any other status or characteristic protected by applicable laws, regulations, andordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to HR@insightglobal.com.