Software Developer

Develop application architecture and distill abstract architecture into software design using a thorough understanding of implementation issues and core development platforms. Design, develop and maintain Hadoop Distributed File System (HDFS) to store and organize data effectively across the cluster by implementing data partitioning, replication strategies, and managing metadata. Develop shell scripts for configuration checks and files transformation. Develop and optimize MapReduce, Yarn, Spark and other processing jobs to transform raw data into usable formats for analytics. Develop SQL queries, optimize performance; create data models and schemas for easier access using data query tools including HBase, Hive, Impala, Kudu and Nifi to query and analyze data stored in Hadoop. Develop real-time streaming applications using Spark Streaming, Structured Streaming and other streaming APIs. Develop the automated testing framework for Junit testing for the developed modules. Conduct design and code review, testing and debugging activities at the application level. Identify and resolve performance bottlenecks in Hadoop jobs, clusters, storage. Optimize job configurations, tuning parameters, and improving overall system efficiency. Design, develop and managing workflows using Oozie XML-based language. Design, develop and maintain data pipelines using DataProc on Google Cloud Platform (GCP) to process large-scale data sets efficiently. Implement data processing workflows using DataProc for tasks. Integrate DataProc with other GCP services including BigQuery, Cloud Storage, Dataflow, and Pub/Sub to build end-to-end data processing solutions. Integrate Spark with BigQuery to export data. Implement CI/CD pipelines for Hadoop applications including automated testing, building, and deployment using tools such as GitHub, Google Cloud Build, Google Artifact Registry and Google Kubernetes Engine (GKE). Develop CI/CD pipelines to build and deploy applications on different compute options. Automate the build process by setting up Cloud Build triggers. Document processes, workflows and configurations to facilitate knowledge sharing.

Work Location: Various unanticipated work locations throughout the United States; relocation may be required. Must be willing to relocate.

Minimum Requirements:

Education: Bachelor -- Computer Science or Electronic Engineering (will accept foreign education equivalent)

Experience: Five (5) years