Data Engineer (1410)

Summary:The Data Engineer is part of a skilled IT team that will work independently and collectively with other engineers to identify and test potential enhancements and resolve technical issues with the rapid scaling of an intelligence application.Essential Functions:Collaboratively develop and implement custom-developed ETL scripts; and support software products as needed.Ensure data quality, consistency, and integrity throughout the pipeline.Create, monitor, and adjust data ingestion flows.Process raw data by transforming it into a usable form (ETL).Rapidly create data pipelines to support training exercise with little prior notification to connect fielded systems to cloud-based infrastructure as well as other fielded systems.Develop for and work with the ZIngest system, an ingestion system developed by Zapata Technology.Perform System Administration functions such as database management, data management, and fixing issues that arise, including the installation of peripheral software and software updates.Implement rapid features and API capabilities to meet evolving customer needs for data access and dissemination.Interface with the customer(s) to create ingestion feeds for unique systems with ability to implement solutions with few prior known details/documentations about said systems.Use source-code control to track and protect changes to the code baseline.Manage containerization efforts (Docker, Podman) and orchestration. Upgrade and migrate applications to containers to allow the required scalability and move legacy monolith applications to a microservices type environment. Document Infrastructure administration, tear down, and creation.Assist with Automation and Security tasks: Implement Infrastructure as Code (IaC) using Puppet, Ansible or similar technology to meet security mandated change management guidelines. Ensure application and database compliance with DISA and GISA standards and regulations. Work with Cyber-Security personnel to document security vulnerabilities and any mitigations to reduce risks that cannot be removed.Job Qualifications:Knowledge of Agile programming and development concepts including Jira or similar.Knowledge of DevSecOps processes including Infrastructure as Code using Puppet, Ansible, etc.Knowledge of containerization and container orchestration including networking using Docker, Podman, or Kubernetes.Experience with the full DataOps lifecycle including ingestion and ETL processes into databases, data lakes, or data warehouses.Detail-oriented and organized; able to understand information systems and ensure accuracy of work.Grasp of Software Engineering concepts, especially scripting in Python and Groovy.Linux System administration knowledge: know the Linux command line, service management, software configuration concepts in lieu of virtual machine constraints.Ability to convert data and fix malformed data. This requires an understanding of XML, JSON, and regular expressions.Preferred Qualifications:An understanding of pub-sub messaging systems including RabbitMQ, Pulsar and Kafka.A solid understanding of and experience in NiFi data flow, from configuration to creating new flows and making new processors.Experience with troubleshooting the NiFi software.Knowledge of NiFi, with ability to keep the cluster running.Ability to write scripts in Groovy and Python within NiFi and Linux.Ability to write new processors in NiFi using Java.Certifications Required:IAT Level III certification is needed for this role, but having a level II with ability to obtain level III in agreed upon time may be accepted.Education/Experience include:A bachelor’s degree with 9 to 12 years of experience in performing similar or related work is required. Four years of experience may be substituted for a bachelor’s degree.Clearance Type:TS/SCI clearance required.Travel:Travel may be required for this role but estimated at 2 weeks or less per TDY and an average of 1-3 times per year.AAP/EEO Statement:Zapata Technology is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, genetic information, creed, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.AAP/EEO Employer: Minorities/Females/Disabled/Veterans

#J-18808-Ljbffr