Tesla
Software Engineer, Generalist, AI Infrastructure
Tesla, Palo Alto, California, United States, 94306
As a Software Engineer within Autopilot, you will work on reinforcing, optimizing, and scaling our neural network training infrastructure.
At the core of our self-driving capabilities, there are different neural networks that the Deep Learning team is designing to train large amounts of data. Robustly training jobs at scale, should it be for production models or quick experiments, and completing them in the shortest amount of time possible, is critical to our mission
Responsibilities
Write robust Python software code in our machine learning training repository while applying best software practices to support machine learning scientists in tasks such as fetching training data, preprocessing it, and orchestrating the training runsIntegrate the training software into our continuous integration cluster to support metrics persistence across experiments, weekly/nightly neural network builds, and other unit / throughput testsProfile performance of training software in our training cluster, identify bottlenecks in and between CPU/GPU code execution, and work on optimizing its throughput and scalability within and across nodes to ultimately reduce convergence timeCoordinate with the team managing the hardware cluster to maintain high availability / jobs throughput for Machine LearningRequirements
Practical experience programming in Python and/or C/C++Proficient in system-level software, in particular hardware-software interactions and resource utilizationUnderstanding of modern machine learning concepts and state of the art deep learningExperience working with training frameworks, ideally PyTorchDemonstrated experience scaling neural network training jobs across clusters of GPU'sExperience programming in CudaProfiling and optimizing CPU-GPU interactions (pipelining compute/transfers, etc.)Devops experience, in particular dealing with clusters of training nodes, and filesystems for very large amount of training dataCompensation and BenefitsBenefits
Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire:
Aetna PPO and HSA plans > 2 medical plan options with $0 payroll deductionFamily-building, fertility, adoption and surrogacy benefitsDental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contributionCompany Paid (Health Savings Account) HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSAHealthcare and Dependent Care Flexible Spending Accounts (FSA)LGBTQ+ care concierge services401(k) with employer match, Employee Stock Purchase Plans, and other financial benefitsCompany paid Basic Life, AD&D, short-term and long-term disability insuranceEmployee Assistance ProgramSick and Vacation time (Flex time for salary positions), and Paid HolidaysBack-up childcare and parenting support resourcesVoluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insuranceWeight Loss and Tobacco Cessation ProgramsTesla Babies programCommuter benefitsEmployee discounts and perks programExpected Compensation
$104,000 - $360,000/annual salary, depending on level + cash and stock awards + benefits
Pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.
At the core of our self-driving capabilities, there are different neural networks that the Deep Learning team is designing to train large amounts of data. Robustly training jobs at scale, should it be for production models or quick experiments, and completing them in the shortest amount of time possible, is critical to our mission
Responsibilities
Write robust Python software code in our machine learning training repository while applying best software practices to support machine learning scientists in tasks such as fetching training data, preprocessing it, and orchestrating the training runsIntegrate the training software into our continuous integration cluster to support metrics persistence across experiments, weekly/nightly neural network builds, and other unit / throughput testsProfile performance of training software in our training cluster, identify bottlenecks in and between CPU/GPU code execution, and work on optimizing its throughput and scalability within and across nodes to ultimately reduce convergence timeCoordinate with the team managing the hardware cluster to maintain high availability / jobs throughput for Machine LearningRequirements
Practical experience programming in Python and/or C/C++Proficient in system-level software, in particular hardware-software interactions and resource utilizationUnderstanding of modern machine learning concepts and state of the art deep learningExperience working with training frameworks, ideally PyTorchDemonstrated experience scaling neural network training jobs across clusters of GPU'sExperience programming in CudaProfiling and optimizing CPU-GPU interactions (pipelining compute/transfers, etc.)Devops experience, in particular dealing with clusters of training nodes, and filesystems for very large amount of training dataCompensation and BenefitsBenefits
Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire:
Aetna PPO and HSA plans > 2 medical plan options with $0 payroll deductionFamily-building, fertility, adoption and surrogacy benefitsDental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contributionCompany Paid (Health Savings Account) HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSAHealthcare and Dependent Care Flexible Spending Accounts (FSA)LGBTQ+ care concierge services401(k) with employer match, Employee Stock Purchase Plans, and other financial benefitsCompany paid Basic Life, AD&D, short-term and long-term disability insuranceEmployee Assistance ProgramSick and Vacation time (Flex time for salary positions), and Paid HolidaysBack-up childcare and parenting support resourcesVoluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insuranceWeight Loss and Tobacco Cessation ProgramsTesla Babies programCommuter benefitsEmployee discounts and perks programExpected Compensation
$104,000 - $360,000/annual salary, depending on level + cash and stock awards + benefits
Pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.