Tesla
HPC Engineer, AI Infrastructure
Tesla, Stanford, California, United States, 94305
Tesla's Supercomputing/AI infrastructure team works directly with the high-performance computing and machine learning infrastructure on which our ML algorithms run; this includes virtual simulations, Autopilot hardware, silicon design, and Dojo. With the rapidly-growing need for more data and optimized compute resources, cluster builds are getting larger and increasingly complex. Continued development/automation of deployment, monitoring, self-healing and alerting processes is imperative to the success of our engineering groups. As the scope and impact of our Tesla Bot, Full-Self-Driving (FSD) & Robotaxi efforts continue to scale, so does the value of this team and its work.
As an HPC Engineer, you will be responsible for maintaining and improving our platform to ensure our Full-Self-Driving (FSD), Tesla Bot & Dojo engineering teams have the necessary tools and resources to be productive. This includes managing/operating our AI infrastructure, monitoring compute/GPU/network metrics, Linux troubleshooting & performance tuning, and collaborating with our Data Center team to coordinate the smooth operation of hundreds of servers & bring up new GPU capacity. Your work will directly facilitate neural network training at scale, streamline FSD development, and enable Dojo to become the most powerful supercomputer to date.
Responsibilities
Support the AI/ML cluster infrastructure on both GPU and Dojo platforms, focusing on systems automation, configuration management and deployment at scaleImprove our monitoring & self-healing pipelines, as well as security postureWork with hardware and storage vendors to tune and optimize our server, storage and network performancePerformance tuning & OS provisioning on Linux systemsManage HPC clusters, workloads and applicationsAutomation and systems engineeringParticipate in 24x7 on-call rotationRequirements
Proficiency with scripting languages such as Python or BashProficiency with Linux & network fundamentalsExperience with configuration management software (Ansible, etc.), systems monitoring & alerting (Prometheus, Grafana, Telegraf, Splunk, etc.) is a plusExperience with high-throughput low-latency networks, GPU-based computing systems, and/or high performance storage systems is a plusExperience with Slurm, LSF and storage management of parallel file systems is a plusBachelor's Degree in Computer Science, Computer Engineering, Electrical Engineering, Physics or proof of exceptional skills in related field3+ years of additional equivalent experience or evidence of exceptional ability related to the positionCompensation and BenefitsBenefits
Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire:
Aetna PPO and HSA plans > 2 medical plan options with $0 payroll deductionFamily-building, fertility, adoption and surrogacy benefitsDental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contributionCompany Paid (Health Savings Account) HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSAHealthcare and Dependent Care Flexible Spending Accounts (FSA)LGBTQ+ care concierge services401(k) with employer match, Employee Stock Purchase Plans, and other financial benefitsCompany paid Basic Life, AD&D, short-term and long-term disability insuranceEmployee Assistance ProgramSick and Vacation time (Flex time for salary positions), and Paid HolidaysBack-up childcare and parenting support resourcesVoluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insuranceWeight Loss and Tobacco Cessation ProgramsTesla Babies programCommuter benefitsEmployee discounts and perks programExpected Compensation
$120,000 - $300,000/annual salary + cash and stock awards + benefits
Pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.
As an HPC Engineer, you will be responsible for maintaining and improving our platform to ensure our Full-Self-Driving (FSD), Tesla Bot & Dojo engineering teams have the necessary tools and resources to be productive. This includes managing/operating our AI infrastructure, monitoring compute/GPU/network metrics, Linux troubleshooting & performance tuning, and collaborating with our Data Center team to coordinate the smooth operation of hundreds of servers & bring up new GPU capacity. Your work will directly facilitate neural network training at scale, streamline FSD development, and enable Dojo to become the most powerful supercomputer to date.
Responsibilities
Support the AI/ML cluster infrastructure on both GPU and Dojo platforms, focusing on systems automation, configuration management and deployment at scaleImprove our monitoring & self-healing pipelines, as well as security postureWork with hardware and storage vendors to tune and optimize our server, storage and network performancePerformance tuning & OS provisioning on Linux systemsManage HPC clusters, workloads and applicationsAutomation and systems engineeringParticipate in 24x7 on-call rotationRequirements
Proficiency with scripting languages such as Python or BashProficiency with Linux & network fundamentalsExperience with configuration management software (Ansible, etc.), systems monitoring & alerting (Prometheus, Grafana, Telegraf, Splunk, etc.) is a plusExperience with high-throughput low-latency networks, GPU-based computing systems, and/or high performance storage systems is a plusExperience with Slurm, LSF and storage management of parallel file systems is a plusBachelor's Degree in Computer Science, Computer Engineering, Electrical Engineering, Physics or proof of exceptional skills in related field3+ years of additional equivalent experience or evidence of exceptional ability related to the positionCompensation and BenefitsBenefits
Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire:
Aetna PPO and HSA plans > 2 medical plan options with $0 payroll deductionFamily-building, fertility, adoption and surrogacy benefitsDental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contributionCompany Paid (Health Savings Account) HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSAHealthcare and Dependent Care Flexible Spending Accounts (FSA)LGBTQ+ care concierge services401(k) with employer match, Employee Stock Purchase Plans, and other financial benefitsCompany paid Basic Life, AD&D, short-term and long-term disability insuranceEmployee Assistance ProgramSick and Vacation time (Flex time for salary positions), and Paid HolidaysBack-up childcare and parenting support resourcesVoluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insuranceWeight Loss and Tobacco Cessation ProgramsTesla Babies programCommuter benefitsEmployee discounts and perks programExpected Compensation
$120,000 - $300,000/annual salary + cash and stock awards + benefits
Pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.