Lead High Performance Computing (HPC) Engineer

Job SummaryThe CoreHPC team at UCSF is seeking a Lead HPC Engineer to fill a technical leadership role in the design, development, maintenance, and day-to-day operations of the new CoreHPC cluster. This next-generation institutional HPC cluster is focused on AI and Data Science, which is currently being built.The Lead HPC Engineer will:Apply advanced systems infrastructure concepts and skills to the operations and improvement of large-scale and highly complex research Cyber Infrastructure (CI) with unique computing, networking, and storage systems designed to address cutting-edge research problems. Apply their engineering and design skills to develop new CI solutions and to develop and enhance monitoring to maintain the integrity of CI systems.Select methods, techniques, and evaluation criteria to develop new CI solutions to address complex research problems.Be an active member of the support and maintenance efforts for the CoreHPC cluster, resolving user issues, fixing technical problems, resolving outages, patching, and maintaining systems uptime and availability.Provides consultation, support, and guidance to researchers on how to address computational problems using standard tools, packages, and approaches.Develop enhancements of monitoring to maintain the integrity of CI systems.Participate in multiple technical projects simultaneously.Applies working knowledge of security control frameworks to maintain the integrity of the CI systems and the research being performed on them.Gives presentations to associated teams and other technical units.Evaluates new technologies including performing moderate to complex cost/benefit analyses.This position may lead cross-functional technical working groups and projects in support of onboarding research customers, or making systems improvements. May lead a team of systems/infrastructure professionals.The final salary and offer components are subject to additional approvals based on UC policy.our placement within the salary range is dependent on a number of factors including your work experience and internal equity within this position classification at UCSF. For positions that are represented by a labor union, placement within the salary range will be guided by the rules in the collective bargaining agreement.The salary range for this position is $132,400 - $198,600 (Annual Rate).To learn more about the benefits of working at UCSF, including total compensation, please visit: Department DescriptionAcademic Research Systems (ARS) serves the needs of the UCSF research community by providing an integrated repository of HIPAA compliant clinical and life sciences data and a centralized, secure, professionally managed infrastructure for the storage and management of research data. ARS empowers medical scientific investigations by offering secure computing environments, data capture, management and analysis tools, and support services which meet researchers’ needs.The Core HPC team of the Academic Research Service (ARS) focuses on large scale, high performance computational and storage services for UCSF researchers so they can address complex computational, AI,  and data science problems.Required QualificationsBachelor's degree in a related area such as computer science or engineering, and 6+ years of experience with large-scale or HPC systems * or*  10+ years of related experience with large-scale or High-Performance Computing (HPC) systemsKnowledge of HPC job scheduler system design and operation such as SLURM or PBS,Demonstrated skill (5 years +) deploying, managing, and troubleshooting Warewulf (or similar) infiniband-based clustersExpert knowledge of HPC systems infrastructure designStrong knowledge of High-performance parallel filesystems and storage such as GPFS, Lustre, Vast, DDN, etcAdvanced knowledge of computer security best practices and policies including demonstrated experience securing research cyberinfrastructure systems to meet NIST 800-171 / 800-223, HIPPA or IS-3 requirementsAbility to write technical documentation in a clear and concise manner. Ability to develop Req runbooks defining complex technical processes in a clear and concise mannerDemonstrated testing and test planning skills. Demonstrated ability to create automated Req testing.Ability to elicit and communicate technical and non-technical information in a clear and concise manner.Self-motivated and works independently and as part of a team. Demonstrates problem-solving skills. Able to learn effectively and meet deadlines.Understanding of system performance monitoring and actions that can be taken to improve or correct performance.Demonstrated advanced knowledge, skills, and abilities associated with system problem identification and resolution. Experience with design, configuration, operation, repair, and tuning of technology systems.Advanced experience writing and editing the most complex scripts used to perform system maintenance and administration.Preferred QualificationsKnowledge of the design, development, and application of technology and systems to meet business needs.General knowledge of other areas of IT. Thorough understanding of and experience with systems-related issues and actions that can be taken to improve or correct performance.Demonstrated skills associated with adapting equipment and technology to serve user needs. Demonstrated comprehensive understanding of how system management actions affect other systems, system users, and dependent/related functions.About UCSFThe University of California, San Francisco (UCSF) is a leading university dedicated to promoting health worldwide through advanced biomedical research, graduate-level education in the life sciences and health professions, and excellence in patient care. It is the only campus in the 10-campus UC system dedicated exclusively to the health sciences. We bring together the world’s leading experts in nearly every area of health. We are home to five Nobel laureates who have advanced the understanding of cancer, neurodegenerative diseases, aging and stem cells.Pride ValuesUCSF is a diverse community made of people with many skills and talents. We seek candidates whose work experience or community service has prepared them to contribute to our commitment to professionalism, respect, integrity, diversity and excellence – also known as our PRIDE values. In addition to our PRIDE values, UCSF is committed to equity – both in how we deliver care as well as our workforce. We are committed to building a broadly diverse community, nurturing a culture that is welcoming and supportive, and engaging diverse ideas for the provision of culturally competent education, discovery, and patient care. Additional information about UCSF is available at diversity.ucsf.edu Join us to find a rewarding career contributing to improving healthcare worldwide.Equal Employment OpportunityThe University of California San Francisco is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, protected veteran or disabled status, or genetic information.OrganizationCampusJob Code and Payroll Title000520 SYS ADM 4 Job CategoryClinical Systems / IT Professionals Bargaining Unit99 - Policy-Covered (No Bargaining Unit) Employee ClassCareerPercentage100%LocationSan Francisco, CA ShiftDaysShift Length8 Hours Department Name: IT EDW OperationsPosition Type: Full TimeReq Number: 81014BR