Aitopics
Principal Software Engineer - Data Engineering
Aitopics, San Francisco, California, United States, 94199
Principal Software Engineer - Data Engineering
About project44:Transform your supply chain with real-time visibility, automation, and AI-powered insights from Movement by project44. We revolutionize supply chains with our High-Velocity Supply Chain Platform, optimizing global product movement and delivering unparalleled resiliency and value for our customers. We operate the world's most trusted end-to-end visibility platform, tracking over 1 billion shipments annually for 1,300 leading brands across various industries.About the role:As a Principal Software Engineer - Data Engineering at project44, you'll work on the latest technologies to streamline Machine Learning & AI Operations, build scalable data infrastructure, and democratize data access.What you'll be doing:Work with software architecture and design, leveraging best practices from distributed systems, databases, and data platforms.Provide guidance on new technologies and continuous improvement in best practices.Build systems in a multi-cloud environment (AWS & GCP preferred).Build complex metrics solutions with data visualization support for actionable business insights.Leverage expertise in Gen AI tools & methodologies to architect automated data access solutions.Design and develop ETL/ELT processes using Python/Java with Snowflake, Postgres, and other data stores.Knowledge in Data Warehouse/Data Mart design and implementation.Build distributed, reusable, and efficient backend ETLs, implementing security and data protection.Understand automated processes for building, testing, documenting, and deploying applications at scale.Collaborate with insights and data science teams to provide technical solutions and implement new features and data pipelines.Establish quality processes to deliver stable and reliable solutions.Write complex SQL and stored procedures in Snowflake, Postgres, BigQuery.Prepare documentation for all projects.Coach junior team members and contribute to tooling, documentation, and development practices.You could be a great fit if you have:Experience & Education:8+ years of experience in leading Data Engineering efforts.3+ years of experience with Snowflake, Oracle, and NoSQL databases like MongoDB.3+ years of experience in Python/Java.3+ years of experience in ETL development with knowledge of data processing tools.4+ years of experience in data engineering and operations.Experience delivering software solutions in distributed systems.Experience with Neural networks and Gen AI methodologies.Strong experience in building data warehouse solutions and Data Modeling.Strong ETL performance-tuning skills.Experience with ETL, GCP, Unix/Linux, and Git.Experience with PII redaction for ETL pipelines.Expertise in operational data stores and real-time data integration.Bachelor's Degree in computer science or equivalent experience.Technical Skills:Strong programming/scripting knowledge in building and maintaining ETL using Java, SQL, Python, Bash, Go.In-depth knowledge of public clouds - GCP, AWS, PostgreSQL, ElasticSearch, MongoDB, Snowflake, BigQuery.Participate in an on-call rotation for data pipeline failures.Strong experience with Kafka or equivalent event/streaming systems.Experience with Docker, Kubernetes.Experience with RAG, Vector DB, embeddings.Develop and deploy CICD pipelines for Data Engineering.Optimize database performance and capacity utilization.Proficiency with high volume OLTP databases and large data warehouse environments.Understanding of Agile implementation for Data Warehouse Development.Professional Skills/Competency:Focus on developing frameworks for repeatable and scalable solutions.Excellent communication and interpersonal skills.Initiative to recommend and develop innovative approaches.Team player who encourages collaboration.Diversity & Inclusion:At project44, we're designing the future of global supply chains. We are committed to building diverse teams that reflect our communities and creating an inclusive environment where all team members can thrive.project44 is an equal opportunity employer. If you share our values and passion for transforming the way the world moves, we’d love to review your application!
#J-18808-Ljbffr
About project44:Transform your supply chain with real-time visibility, automation, and AI-powered insights from Movement by project44. We revolutionize supply chains with our High-Velocity Supply Chain Platform, optimizing global product movement and delivering unparalleled resiliency and value for our customers. We operate the world's most trusted end-to-end visibility platform, tracking over 1 billion shipments annually for 1,300 leading brands across various industries.About the role:As a Principal Software Engineer - Data Engineering at project44, you'll work on the latest technologies to streamline Machine Learning & AI Operations, build scalable data infrastructure, and democratize data access.What you'll be doing:Work with software architecture and design, leveraging best practices from distributed systems, databases, and data platforms.Provide guidance on new technologies and continuous improvement in best practices.Build systems in a multi-cloud environment (AWS & GCP preferred).Build complex metrics solutions with data visualization support for actionable business insights.Leverage expertise in Gen AI tools & methodologies to architect automated data access solutions.Design and develop ETL/ELT processes using Python/Java with Snowflake, Postgres, and other data stores.Knowledge in Data Warehouse/Data Mart design and implementation.Build distributed, reusable, and efficient backend ETLs, implementing security and data protection.Understand automated processes for building, testing, documenting, and deploying applications at scale.Collaborate with insights and data science teams to provide technical solutions and implement new features and data pipelines.Establish quality processes to deliver stable and reliable solutions.Write complex SQL and stored procedures in Snowflake, Postgres, BigQuery.Prepare documentation for all projects.Coach junior team members and contribute to tooling, documentation, and development practices.You could be a great fit if you have:Experience & Education:8+ years of experience in leading Data Engineering efforts.3+ years of experience with Snowflake, Oracle, and NoSQL databases like MongoDB.3+ years of experience in Python/Java.3+ years of experience in ETL development with knowledge of data processing tools.4+ years of experience in data engineering and operations.Experience delivering software solutions in distributed systems.Experience with Neural networks and Gen AI methodologies.Strong experience in building data warehouse solutions and Data Modeling.Strong ETL performance-tuning skills.Experience with ETL, GCP, Unix/Linux, and Git.Experience with PII redaction for ETL pipelines.Expertise in operational data stores and real-time data integration.Bachelor's Degree in computer science or equivalent experience.Technical Skills:Strong programming/scripting knowledge in building and maintaining ETL using Java, SQL, Python, Bash, Go.In-depth knowledge of public clouds - GCP, AWS, PostgreSQL, ElasticSearch, MongoDB, Snowflake, BigQuery.Participate in an on-call rotation for data pipeline failures.Strong experience with Kafka or equivalent event/streaming systems.Experience with Docker, Kubernetes.Experience with RAG, Vector DB, embeddings.Develop and deploy CICD pipelines for Data Engineering.Optimize database performance and capacity utilization.Proficiency with high volume OLTP databases and large data warehouse environments.Understanding of Agile implementation for Data Warehouse Development.Professional Skills/Competency:Focus on developing frameworks for repeatable and scalable solutions.Excellent communication and interpersonal skills.Initiative to recommend and develop innovative approaches.Team player who encourages collaboration.Diversity & Inclusion:At project44, we're designing the future of global supply chains. We are committed to building diverse teams that reflect our communities and creating an inclusive environment where all team members can thrive.project44 is an equal opportunity employer. If you share our values and passion for transforming the way the world moves, we’d love to review your application!
#J-18808-Ljbffr