Research Engineer, ML Systems (All Industry Levels)

Joining us as a Research Engineer on the ML Systems team, you’ll be working on cutting-edge ML training and inference systems, optimizing the performance and efficiency of our GPU clusters, and developing new technologies that fine-tune leading consumer AI models with a data flywheel, and serve 20K+ QPS in production with LLMs. Your work will directly contribute to our groundbreaking advancements in AI, helping shape an era where technology is not just a tool, but a companion in our daily lives. At Character.AI, your talent, creativity, and expertise will not just be valued—they will be the catalyst for change in an AI-driven future.About the roleThe ML Systems team is responsible for the research and deployment of systems that efficiently utilize GPU for AI-enabled products. As a research engineer, you will work across teams and our technical stack to improve our training performance and inference runtime. You will get to shape the conversational experience of millions of users per day.Example projects:Write efficient Triton kernels and tune them for our specific models and hardwareDevelop prefix-aware routing algorithms to improve serving cache hit rateTrain and distill LLMs to improve latency while preserving accuracy and engagementsBuild an efficient and scalable distributed RLHF stack powering the model innovationsDevelop systems for efficient multimodal (image gen/video gen) model training & inferenceJob Requirements"All Industry Levels": at least PhD (or equivalent) research experienceWrite clear and clean production system codeStrong understanding of modern machine learning techniques (reinforcement learning, transformers, etc)Track record of exceptional research or creative ML systems projectsComfortable writing model development code (PyTorch) for either training or inferenceNice to HaveExperience training large models in a distributed setting utilizing PyTorch distributed, DeepSpeed, Megatron.Experience working with GPUs & collectives (training, serving, debugging) and writing kernels (Triton, CUDA, CUTLASS).Experience with LLM inference systems and literature such as vLLM and FlashAttention.Familiarity with ML deployment and orchestration (Kubernetes, Docker, cloud)Publications in relevant academic journals or conferences in the field of machine learning and systemsAbout Character.AIFounded in 2021, Character is a leading AI company offering personalized experiences through customizable AI 'Characters.' As one of the most widely used AI platforms worldwide, Character enables users to interact with AI tailored to their unique needs and preferences.In just two years, we achieved unicorn status and were named Google Play's AI App of the Year – a testament to our groundbreaking technology and vision.Ready to shape the future of Consumer AI? At Character, we value diversity and welcome applicants from all backgrounds. As an equal opportunity employer, we firmly uphold a non-discrimination policy based on race, religion, national origin, gender, sexual orientation, age, veteran status, or disability. Your unique perspectives are vital to our success.Compensation Range: $225K - $400KLocationMenlo Park or New York CityTypeFull timeDepartmentTechnical Staff - MLCompensation$225K – $400K • Offers Equity Top-notch health coverage for you & your family, with majority of the premium covered We invest in your future with a generous 401(K) contribution New parents, we've got you covered with incredible paid leave -up to 20 weeks 4 weeks of PTO to explore, unwind & come back recharged️ Daily in-office catering plus a monthly Doordash stipend to help keep you fueled no matter where you are Monthly wellness stipend to support you in your health journey